[==========] Running 17 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from DiskFailure/DiskFailureITest
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0
2026-08-08T01:47:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:47:48Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260808 01:47:48.897825 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41709
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:41709
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:47:49.227913 30427 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:47:49.228652 30427 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:47:49.229125 30427 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:47:49.262773 30427 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:47:49.263231 30427 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:47:49.263520 30427 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:47:49.263792 30427 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:47:49.264060 30427 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:47:49.302649 30427 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:41709
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:47:49.304757 30427 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:47:49.307430 30427 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:49.323880 30433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:50.726293 30432 debug-util.cc:398] Leaking SignalData structure 0x7b08000290e0 after lost signal to thread 30427
W20260808 01:47:49.323977 30435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:51.255978 30427 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.933s user 0.663s sys 1.269s
W20260808 01:47:51.256467 30427 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.933s user 0.663s sys 1.270s
W20260808 01:47:51.256711 30436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1931 milliseconds
W20260808 01:47:51.258226 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:51.258251 30427 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:47:51.262445 30427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:47:51.265872 30427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:47:51.267376 30427 hybrid_clock.cc:648] HybridClock initialized: now 1786153671267320 us; error 94 us; skew 500 ppm
I20260808 01:47:51.268508 30427 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:51.276101 30427 webserver.cc:533] Webserver started at http://127.29.180.126:46449/ using document root <none> and password file <none>
I20260808 01:47:51.278512 30427 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:51.278844 30427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:51.279713 30427 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:51.285104 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.286790 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.287835 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.288767 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.289754 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.290668 30427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.312530 30427 fs_manager.cc:696] Time spent creating directory manager: real 0.021s user 0.013s sys 0.008s
I20260808 01:47:51.317837 30427 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s
I20260808 01:47:51.318167 30427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "492e41c2a19d4115953dbd988a8098e9"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-p3t5"
I20260808 01:47:51.318641 30427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:47:51.384280 30427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:47:51.386138 30427 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:47:51.386713 30427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:51.476745 30427 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:41709
I20260808 01:47:51.476876 30498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:41709 every 8 connection(s)
I20260808 01:47:51.480296 30427 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:47:51.490942 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 30427
I20260808 01:47:51.491490 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:47:51.516924 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9: Bootstrap starting.
I20260808 01:47:51.525492 30500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:51.528192 30500 log.cc:826] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9: Log is configured to *not* fsync() on all Append() calls
I20260808 01:47:51.534291 30500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9: No bootstrap required, opened a new log
I20260808 01:47:51.562223 30500 raft_consensus.cc:359] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } }
I20260808 01:47:51.563275 30500 raft_consensus.cc:385] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:51.563598 30500 raft_consensus.cc:740] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492e41c2a19d4115953dbd988a8098e9, State: Initialized, Role: FOLLOWER
I20260808 01:47:51.564816 30500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } }
I20260808 01:47:51.565502 30500 raft_consensus.cc:399] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:47:51.565884 30500 raft_consensus.cc:493] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:47:51.566293 30500 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:51.571038 30500 raft_consensus.cc:515] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } }
I20260808 01:47:51.572096 30500 leader_election.cc:304] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 492e41c2a19d4115953dbd988a8098e9; no voters:
I20260808 01:47:51.574244 30500 leader_election.cc:290] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:47:51.574635 30506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:51.577915 30506 raft_consensus.cc:697] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [term 1 LEADER]: Becoming Leader. State: Replica: 492e41c2a19d4115953dbd988a8098e9, State: Running, Role: LEADER
I20260808 01:47:51.578887 30506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } }
I20260808 01:47:51.580168 30500 sys_catalog.cc:565] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:47:51.592537 30508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492e41c2a19d4115953dbd988a8098e9. Latest consensus state: current_term: 1 leader_uuid: "492e41c2a19d4115953dbd988a8098e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } } }
I20260808 01:47:51.593325 30508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [sys.catalog]: This master's current role is: LEADER
I20260808 01:47:51.591498 30507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "492e41c2a19d4115953dbd988a8098e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492e41c2a19d4115953dbd988a8098e9" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41709 } } }
I20260808 01:47:51.594993 30507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9 [sys.catalog]: This master's current role is: LEADER
I20260808 01:47:51.598523 30515 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:47:51.614650 30515 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:47:51.636121 30515 catalog_manager.cc:1383] Generated new cluster ID: 3eef40be1bed4f7c895d09813970f1a4
I20260808 01:47:51.636540 30515 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:47:51.656296 30515 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:47:51.658663 30515 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:47:51.686285 30515 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 492e41c2a19d4115953dbd988a8098e9: Generated new TSK 0
I20260808 01:47:51.687666 30515 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:47:51.705708 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--builtin_ntp_servers=127.29.180.84:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:47:52.046222 30525 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:47:52.046830 30525 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:47:52.047340 30525 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:47:52.082517 30525 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:47:52.082970 30525 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:47:52.083772 30525 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:47:52.125172 30525 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:47:52.126669 30525 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:47:52.128525 30525 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:52.143965 30531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:52.148113 30532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:52.150283 30525 server_base.cc:1061] running on GCE node
W20260808 01:47:52.148990 30534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:53.393466 30525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:47:53.396481 30525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:47:53.397992 30525 hybrid_clock.cc:648] HybridClock initialized: now 1786153673397935 us; error 58 us; skew 500 ppm
I20260808 01:47:53.398878 30525 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:53.406658 30525 webserver.cc:533] Webserver started at http://127.29.180.65:41087/ using document root <none> and password file <none>
I20260808 01:47:53.408280 30525 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:53.408545 30525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:53.409276 30525 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:53.414050 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.415266 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.416241 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.417142 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.418115 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.419013 30525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.440941 30525 fs_manager.cc:696] Time spent creating directory manager: real 0.021s user 0.016s sys 0.005s
I20260808 01:47:53.445513 30525 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:53.445837 30525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:47:53.446277 30525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:47:53.520610 30525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:47:53.522341 30525 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:47:53.522820 30525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:53.525913 30525 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:53.530753 30525 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:53.531057 30525 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:53.531345 30525 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:53.531533 30525 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:53.698072 30525 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:42159
I20260808 01:47:53.698253 30660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:42159 every 8 connection(s)
I20260808 01:47:53.700886 30525 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:47:53.709046 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 30525
I20260808 01:47:53.709661 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:47:53.716738 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--builtin_ntp_servers=127.29.180.84:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:47:53.726579 30661 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41709
I20260808 01:47:53.727105 30661 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:53.728310 30661 heartbeater.cc:507] Master 127.29.180.126:41709 requested a full tablet report, sending...
I20260808 01:47:53.731772 30464 ts_manager.cc:194] Registered new tserver with Master: 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159)
I20260808 01:47:53.734300 30464 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:58837
W20260808 01:47:54.056655 30665 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:47:54.057245 30665 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:47:54.057777 30665 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:47:54.092293 30665 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:47:54.092741 30665 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:47:54.093555 30665 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:47:54.133926 30665 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:47:54.135432 30665 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:47:54.137378 30665 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:54.153292 30671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:54.738931 30661 heartbeater.cc:499] Master 127.29.180.126:41709 was elected leader, sending a full tablet report...
W20260808 01:47:54.155009 30672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:55.440415 30674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:55.445689 30665 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.292s user 0.461s sys 0.822s
W20260808 01:47:55.446153 30665 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.292s user 0.465s sys 0.822s
W20260808 01:47:55.446722 30673 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1290 milliseconds
I20260808 01:47:55.446798 30665 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:47:55.448629 30665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:47:55.452040 30665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:47:55.453683 30665 hybrid_clock.cc:648] HybridClock initialized: now 1786153675453638 us; error 45 us; skew 500 ppm
I20260808 01:47:55.455049 30665 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:55.465658 30665 webserver.cc:533] Webserver started at http://127.29.180.66:40665/ using document root <none> and password file <none>
I20260808 01:47:55.468279 30665 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:55.468611 30665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:55.469617 30665 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:55.478608 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.480212 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.481448 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.482679 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.484061 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.485394 30665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.512701 30665 fs_manager.cc:696] Time spent creating directory manager: real 0.026s user 0.017s sys 0.008s
I20260808 01:47:55.517339 30665 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:55.517681 30665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
format_stamp: "Formatted at 2026-08-08 01:47:55 on dist-test-slave-p3t5"
I20260808 01:47:55.518142 30665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:47:55.590646 30665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:47:55.592296 30665 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:47:55.592803 30665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:55.596024 30665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:55.601110 30665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:55.601358 30665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:55.601668 30665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:55.601874 30665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:55.777340 30665 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:45925
I20260808 01:47:55.777464 30798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:45925 every 8 connection(s)
I20260808 01:47:55.780181 30665 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:47:55.783659 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 30665
I20260808 01:47:55.784408 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:47:55.793184 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--builtin_ntp_servers=127.29.180.84:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:47:55.811398 30799 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41709
I20260808 01:47:55.812069 30799 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:55.813759 30799 heartbeater.cc:507] Master 127.29.180.126:41709 requested a full tablet report, sending...
I20260808 01:47:55.816834 30464 ts_manager.cc:194] Registered new tserver with Master: 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:55.818231 30464 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:52693
W20260808 01:47:56.159042 30803 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:47:56.159682 30803 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:47:56.160238 30803 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:47:56.198655 30803 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:47:56.199126 30803 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:47:56.200106 30803 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:47:56.240353 30803 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41709
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:47:56.242012 30803 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:47:56.243731 30803 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:56.257387 30809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:56.821935 30799 heartbeater.cc:499] Master 127.29.180.126:41709 was elected leader, sending a full tablet report...
W20260808 01:47:56.259936 30810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:57.544680 30811 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1283 milliseconds
W20260808 01:47:57.546515 30812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:57.549185 30803 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.290s user 0.001s sys 0.004s
W20260808 01:47:57.549489 30803 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.291s user 0.001s sys 0.004s
I20260808 01:47:57.549765 30803 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:47:57.551021 30803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:47:57.555011 30803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:47:57.556517 30803 hybrid_clock.cc:648] HybridClock initialized: now 1786153677556439 us; error 85 us; skew 500 ppm
I20260808 01:47:57.557518 30803 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:57.565557 30803 webserver.cc:533] Webserver started at http://127.29.180.67:35507/ using document root <none> and password file <none>
I20260808 01:47:57.567186 30803 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:57.567473 30803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:57.568187 30803 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:57.573081 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.574354 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.575357 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.576332 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.577443 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.578692 30803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.603629 30803 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.015s sys 0.009s
I20260808 01:47:57.608518 30803 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:57.608826 30803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "854c7b716e7c4107b4bee282d113f670"
format_stamp: "Formatted at 2026-08-08 01:47:57 on dist-test-slave-p3t5"
I20260808 01:47:57.609262 30803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:47:57.681872 30803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:47:57.683881 30803 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:47:57.684636 30803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:57.687798 30803 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:57.692653 30803 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:57.692914 30803 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:57.693202 30803 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:57.693401 30803 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:47:57.870105 30803 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:37937
I20260808 01:47:57.870291 30935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:37937 every 8 connection(s)
I20260808 01:47:57.872983 30803 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:47:57.883282 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 30803
I20260808 01:47:57.883844 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:47:57.897912 30936 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41709
I20260808 01:47:57.898392 30936 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:57.899507 30936 heartbeater.cc:507] Master 127.29.180.126:41709 requested a full tablet report, sending...
I20260808 01:47:57.902002 30464 ts_manager.cc:194] Registered new tserver with Master: 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:47:57.903708 30464 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:37483
I20260808 01:47:57.907132 30417 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260808 01:47:57.996855 30464 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42796:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:47:58.088330 30869 tablet_service.cc:1511] Processing CreateTablet for tablet 6f57cdc681004e14b9db4f1fa24628e3 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:47:58.090288 30868 tablet_service.cc:1511] Processing CreateTablet for tablet 7cdada0d44314282a8b07ea0fa3d7774 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:47:58.095681 30866 tablet_service.cc:1511] Processing CreateTablet for tablet b15eefc2a48141d3a0700453ac4bbd6d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:47:58.100885 30867 tablet_service.cc:1511] Processing CreateTablet for tablet ecb7b0c5d66e42c4bbc3c3b6c9e3e18d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:47:58.108439 30590 tablet_service.cc:1511] Processing CreateTablet for tablet b15eefc2a48141d3a0700453ac4bbd6d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:47:58.106820 30592 tablet_service.cc:1511] Processing CreateTablet for tablet 7cdada0d44314282a8b07ea0fa3d7774 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:47:58.108057 30593 tablet_service.cc:1511] Processing CreateTablet for tablet 6f57cdc681004e14b9db4f1fa24628e3 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:47:58.106839 30591 tablet_service.cc:1511] Processing CreateTablet for tablet ecb7b0c5d66e42c4bbc3c3b6c9e3e18d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:47:58.110491 30732 tablet_service.cc:1511] Processing CreateTablet for tablet 6f57cdc681004e14b9db4f1fa24628e3 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:47:58.114845 30731 tablet_service.cc:1511] Processing CreateTablet for tablet 7cdada0d44314282a8b07ea0fa3d7774 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:47:58.092921 30870 tablet_service.cc:1511] Processing CreateTablet for tablet 02a3e4affdb54dbeab220185fe1b31d4 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:47:58.118251 30730 tablet_service.cc:1511] Processing CreateTablet for tablet ecb7b0c5d66e42c4bbc3c3b6c9e3e18d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:47:58.118995 30729 tablet_service.cc:1511] Processing CreateTablet for tablet b15eefc2a48141d3a0700453ac4bbd6d (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:47:58.122942 30733 tablet_service.cc:1511] Processing CreateTablet for tablet 02a3e4affdb54dbeab220185fe1b31d4 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:47:58.121984 30594 tablet_service.cc:1511] Processing CreateTablet for tablet 02a3e4affdb54dbeab220185fe1b31d4 (DEFAULT_TABLE table=test-workload [id=f3b5c23934b844d29bab1a97643d0bef]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:47:58.211463 30960 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: Bootstrap starting.
I20260808 01:47:58.240695 30967 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Bootstrap starting.
I20260808 01:47:58.245901 30960 tablet_bootstrap.cc:654] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.248292 30962 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670: Bootstrap starting.
I20260808 01:47:58.247061 30963 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670: Bootstrap starting.
I20260808 01:47:58.253521 30961 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: Bootstrap starting.
I20260808 01:47:58.258453 30960 log.cc:826] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: Log is configured to *not* fsync() on all Append() calls
I20260808 01:47:58.262926 30965 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:47:58.271659 30971 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:47:58.271796 30973 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Bootstrap starting.
I20260808 01:47:58.274780 30962 tablet_bootstrap.cc:654] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.281256 30971 tablet_bootstrap.cc:654] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.281584 30969 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:47:58.287497 30963 tablet_bootstrap.cc:654] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.282857 30975 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:47:58.287811 30960 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: No bootstrap required, opened a new log
I20260808 01:47:58.291457 30965 tablet_bootstrap.cc:654] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.286320 30971 log.cc:826] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Log is configured to *not* fsync() on all Append() calls
I20260808 01:47:58.298151 30960 ts_tablet_manager.cc:1403] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.016s
I20260808 01:47:58.302451 30966 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Bootstrap starting.
I20260808 01:47:58.307119 30964 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Bootstrap starting.
I20260808 01:47:58.295619 30961 tablet_bootstrap.cc:654] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.315973 30970 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Bootstrap starting.
I20260808 01:47:58.322795 30973 tablet_bootstrap.cc:654] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.332696 30968 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Bootstrap starting.
I20260808 01:47:58.340086 30961 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: No bootstrap required, opened a new log
I20260808 01:47:58.343528 30961 ts_tablet_manager.cc:1403] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: Time spent bootstrapping tablet: real 0.090s user 0.015s sys 0.003s
I20260808 01:47:58.357438 30973 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: No bootstrap required, opened a new log
I20260808 01:47:58.358053 30973 ts_tablet_manager.cc:1403] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.009s
I20260808 01:47:58.360234 30972 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:47:58.361338 30964 tablet_bootstrap.cc:654] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.362185 30966 tablet_bootstrap.cc:654] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.362372 30975 tablet_bootstrap.cc:654] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.366106 30964 log.cc:826] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Log is configured to *not* fsync() on all Append() calls
I20260808 01:47:58.369704 30962 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670: No bootstrap required, opened a new log
I20260808 01:47:58.370316 30962 ts_tablet_manager.cc:1403] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670: Time spent bootstrapping tablet: real 0.122s user 0.016s sys 0.028s
I20260808 01:47:58.370994 30970 tablet_bootstrap.cc:654] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.373068 30965 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: No bootstrap required, opened a new log
I20260808 01:47:58.373720 30965 ts_tablet_manager.cc:1403] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.113s user 0.018s sys 0.004s
I20260808 01:47:58.374857 30969 tablet_bootstrap.cc:654] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.380033 30963 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670: No bootstrap required, opened a new log
I20260808 01:47:58.380795 30963 ts_tablet_manager.cc:1403] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.012s
I20260808 01:47:58.385341 30962 raft_consensus.cc:359] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.385951 30973 raft_consensus.cc:359] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.387156 30972 tablet_bootstrap.cc:654] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.386499 30962 raft_consensus.cc:385] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.395910 30971 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: No bootstrap required, opened a new log
I20260808 01:47:58.395805 30962 raft_consensus.cc:740] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.396371 30971 ts_tablet_manager.cc:1403] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.010s
I20260808 01:47:58.397166 30962 consensus_queue.cc:260] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.388087 30960 raft_consensus.cc:359] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.400828 30960 raft_consensus.cc:385] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.401191 30960 raft_consensus.cc:740] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.402022 30960 consensus_queue.cc:260] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.403235 30969 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: No bootstrap required, opened a new log
I20260808 01:47:58.403738 30969 ts_tablet_manager.cc:1403] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.008s
I20260808 01:47:58.387058 30973 raft_consensus.cc:385] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.404733 30973 raft_consensus.cc:740] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.385356 30961 raft_consensus.cc:359] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.405254 30961 raft_consensus.cc:385] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.405543 30961 raft_consensus.cc:740] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.405882 30968 tablet_bootstrap.cc:654] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.405700 30973 consensus_queue.cc:260] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.406338 30961 consensus_queue.cc:260] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.414455 30936 heartbeater.cc:499] Master 127.29.180.126:41709 was elected leader, sending a full tablet report...
I20260808 01:47:58.415498 30960 ts_tablet_manager.cc:1434] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670: Time spent starting tablet: real 0.117s user 0.028s sys 0.011s
I20260808 01:47:58.386499 30963 raft_consensus.cc:359] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.416041 30967 tablet_bootstrap.cc:654] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:58.416357 30963 raft_consensus.cc:385] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.417454 30963 raft_consensus.cc:740] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.423092 30962 ts_tablet_manager.cc:1434] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670: Time spent starting tablet: real 0.052s user 0.009s sys 0.008s
I20260808 01:47:58.435472 30964 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: No bootstrap required, opened a new log
I20260808 01:47:58.436165 30964 ts_tablet_manager.cc:1403] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent bootstrapping tablet: real 0.130s user 0.019s sys 0.006s
I20260808 01:47:58.436323 30965 raft_consensus.cc:359] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.436323 30969 raft_consensus.cc:359] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.437548 30965 raft_consensus.cc:385] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.437548 30969 raft_consensus.cc:385] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.438117 30969 raft_consensus.cc:740] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.438113 30965 raft_consensus.cc:740] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.439649 30969 consensus_queue.cc:260] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.443413 30966 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: No bootstrap required, opened a new log
I20260808 01:47:58.443001 30971 raft_consensus.cc:359] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.443740 30971 raft_consensus.cc:385] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.443944 30966 ts_tablet_manager.cc:1403] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent bootstrapping tablet: real 0.142s user 0.023s sys 0.006s
I20260808 01:47:58.444257 30971 raft_consensus.cc:740] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.445225 30971 consensus_queue.cc:260] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.446223 30967 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6: No bootstrap required, opened a new log
I20260808 01:47:58.446250 30969 ts_tablet_manager.cc:1434] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.042s user 0.005s sys 0.006s
I20260808 01:47:58.420179 30963 consensus_queue.cc:260] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.439649 30965 consensus_queue.cc:260] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.449061 30971 ts_tablet_manager.cc:1434] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.052s user 0.017s sys 0.002s
I20260808 01:47:58.449182 30963 ts_tablet_manager.cc:1434] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260808 01:47:58.435549 30961 ts_tablet_manager.cc:1434] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: Time spent starting tablet: real 0.091s user 0.011s sys 0.007s
I20260808 01:47:58.442348 30973 ts_tablet_manager.cc:1434] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Time spent starting tablet: real 0.084s user 0.015s sys 0.014s
I20260808 01:47:58.453270 30968 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6: No bootstrap required, opened a new log
I20260808 01:47:58.459430 30968 ts_tablet_manager.cc:1403] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent bootstrapping tablet: real 0.128s user 0.016s sys 0.004s
I20260808 01:47:58.461671 30970 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: No bootstrap required, opened a new log
I20260808 01:47:58.462116 30970 ts_tablet_manager.cc:1403] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent bootstrapping tablet: real 0.147s user 0.021s sys 0.005s
W20260808 01:47:58.463712 30662 tablet.cc:2406] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:47:58.465520 30975 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: No bootstrap required, opened a new log
I20260808 01:47:58.465962 30975 ts_tablet_manager.cc:1403] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.183s user 0.020s sys 0.006s
I20260808 01:47:58.466542 30965 ts_tablet_manager.cc:1434] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.092s user 0.024s sys 0.016s
I20260808 01:47:58.468956 30972 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: No bootstrap required, opened a new log
I20260808 01:47:58.469022 30990 raft_consensus.cc:493] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.469365 30972 ts_tablet_manager.cc:1403] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.110s user 0.014s sys 0.015s
I20260808 01:47:58.469013 30975 raft_consensus.cc:359] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.469692 30990 raft_consensus.cc:515] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.470566 30975 raft_consensus.cc:385] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.470912 30975 raft_consensus.cc:740] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.471907 30975 consensus_queue.cc:260] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.472579 30972 raft_consensus.cc:359] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.473363 30972 raft_consensus.cc:385] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.459925 30967 ts_tablet_manager.cc:1403] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent bootstrapping tablet: real 0.220s user 0.007s sys 0.019s
I20260808 01:47:58.476377 30990 leader_election.cc:290] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.478353 30972 raft_consensus.cc:740] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.479203 30972 consensus_queue.cc:260] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.474653 30975 ts_tablet_manager.cc:1434] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 01:47:58.482043 30972 ts_tablet_manager.cc:1434] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20260808 01:47:58.514040 30968 raft_consensus.cc:359] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.514576 30966 raft_consensus.cc:359] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.515161 30968 raft_consensus.cc:385] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.515349 30966 raft_consensus.cc:385] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.515501 30968 raft_consensus.cc:740] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.515647 30966 raft_consensus.cc:740] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.516745 30966 consensus_queue.cc:260] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.517138 30967 raft_consensus.cc:359] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.518291 30967 raft_consensus.cc:385] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.518581 30967 raft_consensus.cc:740] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.519393 30967 consensus_queue.cc:260] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.519747 30994 raft_consensus.cc:493] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.520565 30994 raft_consensus.cc:515] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.523990 30968 consensus_queue.cc:260] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.532230 30964 raft_consensus.cc:359] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.533603 30964 raft_consensus.cc:385] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.534004 30964 raft_consensus.cc:740] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.534786 30994 leader_election.cc:290] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:47:58.535941 30970 raft_consensus.cc:359] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.537595 30970 raft_consensus.cc:385] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:58.544447 30970 raft_consensus.cc:740] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Initialized, Role: FOLLOWER
I20260808 01:47:58.548619 30964 consensus_queue.cc:260] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.550979 30970 consensus_queue.cc:260] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.557153 30614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b15eefc2a48141d3a0700453ac4bbd6d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee" is_pre_election: true
I20260808 01:47:58.568795 30614 raft_consensus.cc:2468] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.573271 30990 raft_consensus.cc:493] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.573912 30990 raft_consensus.cc:515] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.573928 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b15eefc2a48141d3a0700453ac4bbd6d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.575917 30753 raft_consensus.cc:2468] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.580847 30968 ts_tablet_manager.cc:1434] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent starting tablet: real 0.121s user 0.006s sys 0.013s
I20260808 01:47:58.580662 30826 leader_election.cc:304] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 854c7b716e7c4107b4bee282d113f670; no voters:
I20260808 01:47:58.584277 30614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee" is_pre_election: true
I20260808 01:47:58.585348 30614 raft_consensus.cc:2468] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.590412 30991 raft_consensus.cc:2804] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.590998 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.591460 30991 raft_consensus.cc:493] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.591651 30753 raft_consensus.cc:2468] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.591920 30991 raft_consensus.cc:3060] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.581794 30990 leader_election.cc:290] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.593603 30826 leader_election.cc:304] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.595796 30967 ts_tablet_manager.cc:1434] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent starting tablet: real 0.120s user 0.012s sys 0.012s
I20260808 01:47:58.596983 30990 raft_consensus.cc:2804] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.597447 30990 raft_consensus.cc:493] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.597859 30990 raft_consensus.cc:3060] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.600684 30966 ts_tablet_manager.cc:1434] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent starting tablet: real 0.156s user 0.018s sys 0.017s
I20260808 01:47:58.607321 30970 ts_tablet_manager.cc:1434] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent starting tablet: real 0.145s user 0.014s sys 0.009s
I20260808 01:47:58.623466 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670" is_pre_election: true
I20260808 01:47:58.624707 30964 ts_tablet_manager.cc:1434] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6: Time spent starting tablet: real 0.188s user 0.024s sys 0.021s
I20260808 01:47:58.624512 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.625447 30753 raft_consensus.cc:2468] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25643cac26834dbebc98a8630e53e9ee in term 0.
I20260808 01:47:58.628063 30548 leader_election.cc:304] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.629904 30994 raft_consensus.cc:2804] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.629248 30991 raft_consensus.cc:515] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.630407 30994 raft_consensus.cc:493] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.630905 30994 raft_consensus.cc:3060] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.630183 30990 raft_consensus.cc:515] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
W20260808 01:47:58.634439 30937 tablet.cc:2406] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:47:58.639086 30890 raft_consensus.cc:2393] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 25643cac26834dbebc98a8630e53e9ee in current term 1: Already voted for candidate 854c7b716e7c4107b4bee282d113f670 in this term.
I20260808 01:47:58.644965 30614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee"
I20260808 01:47:58.646942 30613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b15eefc2a48141d3a0700453ac4bbd6d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee"
I20260808 01:47:58.647025 30752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b15eefc2a48141d3a0700453ac4bbd6d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
I20260808 01:47:58.647919 30613 raft_consensus.cc:3060] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.648135 30752 raft_consensus.cc:3060] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.642416 30990 leader_election.cc:290] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Requested vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.642539 30991 leader_election.cc:290] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Requested vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.650745 31005 raft_consensus.cc:493] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.651221 31005 raft_consensus.cc:515] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.645228 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
I20260808 01:47:58.654505 30753 raft_consensus.cc:3060] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.653328 30994 raft_consensus.cc:515] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.663185 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.664821 30994 leader_election.cc:290] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 election: Requested vote from peers 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:47:58.664676 30751 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25643cac26834dbebc98a8630e53e9ee in term 0.
I20260808 01:47:58.667033 30614 raft_consensus.cc:2393] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 854c7b716e7c4107b4bee282d113f670 in current term 1: Already voted for candidate 25643cac26834dbebc98a8630e53e9ee in this term.
I20260808 01:47:58.668063 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670"
I20260808 01:47:58.669287 30890 raft_consensus.cc:2393] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 25643cac26834dbebc98a8630e53e9ee in current term 1: Already voted for candidate 854c7b716e7c4107b4bee282d113f670 in this term.
I20260808 01:47:58.669317 30548 leader_election.cc:304] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.669821 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecb7b0c5d66e42c4bbc3c3b6c9e3e18d" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
I20260808 01:47:58.670009 30752 raft_consensus.cc:2468] T b15eefc2a48141d3a0700453ac4bbd6d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.674328 31005 leader_election.cc:290] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:47:58.674680 30826 leader_election.cc:304] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.676095 30990 raft_consensus.cc:2804] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:58.675801 30753 raft_consensus.cc:2468] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.677053 30990 raft_consensus.cc:697] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Becoming Leader. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Running, Role: LEADER
I20260808 01:47:58.675141 30994 raft_consensus.cc:2804] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.677680 30994 raft_consensus.cc:493] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.678203 30994 raft_consensus.cc:3060] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.677937 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670" is_pre_election: true
I20260808 01:47:58.677794 30826 leader_election.cc:304] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters: 25643cac26834dbebc98a8630e53e9ee
I20260808 01:47:58.676649 30613 raft_consensus.cc:2468] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.673077 30548 leader_election.cc:304] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee; no voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6
I20260808 01:47:58.678838 30990 consensus_queue.cc:237] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.681118 30991 raft_consensus.cc:2804] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:58.681166 31005 raft_consensus.cc:2749] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:47:58.681943 30991 raft_consensus.cc:697] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Becoming Leader. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Running, Role: LEADER
I20260808 01:47:58.682922 30991 consensus_queue.cc:237] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.680330 30890 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25643cac26834dbebc98a8630e53e9ee in term 0.
I20260808 01:47:58.688757 30994 raft_consensus.cc:515] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.694316 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
I20260808 01:47:58.695302 30751 raft_consensus.cc:3060] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.702387 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "25643cac26834dbebc98a8630e53e9ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670"
I20260808 01:47:58.703402 30890 raft_consensus.cc:3060] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.694182 30994 leader_election.cc:290] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 election: Requested vote from peers 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:47:58.711884 30991 raft_consensus.cc:493] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.712589 30991 raft_consensus.cc:515] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.719215 30613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cdada0d44314282a8b07ea0fa3d7774" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee" is_pre_election: true
I20260808 01:47:58.720429 30613 raft_consensus.cc:2468] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.721573 30991 leader_election.cc:290] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.724644 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cdada0d44314282a8b07ea0fa3d7774" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.725129 30826 leader_election.cc:304] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 854c7b716e7c4107b4bee282d113f670; no voters:
I20260808 01:47:58.725888 30753 raft_consensus.cc:2468] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.726699 30991 raft_consensus.cc:493] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:58.728713 30987 raft_consensus.cc:2804] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.729367 30987 raft_consensus.cc:493] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.729610 30751 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25643cac26834dbebc98a8630e53e9ee in term 1.
I20260808 01:47:58.729995 30987 raft_consensus.cc:3060] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.731480 30548 leader_election.cc:304] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.733158 30994 raft_consensus.cc:2804] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:58.733960 30994 raft_consensus.cc:697] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 1 LEADER]: Becoming Leader. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Running, Role: LEADER
I20260808 01:47:58.727504 30991 raft_consensus.cc:515] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.735973 30994 consensus_queue.cc:237] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.738370 30991 leader_election.cc:290] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.742413 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a3e4affdb54dbeab220185fe1b31d4" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
I20260808 01:47:58.743283 30751 raft_consensus.cc:2468] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.742139 30463 catalog_manager.cc:5697] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854c7b716e7c4107b4bee282d113f670 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:47:58.747336 30613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a3e4affdb54dbeab220185fe1b31d4" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee" is_pre_election: true
I20260808 01:47:58.748548 30613 raft_consensus.cc:2468] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 0.
I20260808 01:47:58.753225 30826 leader_election.cc:304] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25643cac26834dbebc98a8630e53e9ee, 854c7b716e7c4107b4bee282d113f670; no voters:
I20260808 01:47:58.757257 30991 raft_consensus.cc:2804] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:58.757802 30991 raft_consensus.cc:493] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:58.758247 30991 raft_consensus.cc:3060] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.774355 30464 catalog_manager.cc:5697] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 25643cac26834dbebc98a8630e53e9ee (127.29.180.65). New cstate: current_term: 1 leader_uuid: "25643cac26834dbebc98a8630e53e9ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:47:58.779237 30890 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25643cac26834dbebc98a8630e53e9ee in term 1.
I20260808 01:47:58.785975 30987 raft_consensus.cc:515] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.793776 30991 raft_consensus.cc:515] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.795840 30613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cdada0d44314282a8b07ea0fa3d7774" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee"
I20260808 01:47:58.796638 30613 raft_consensus.cc:3060] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.800987 30614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a3e4affdb54dbeab220185fe1b31d4" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25643cac26834dbebc98a8630e53e9ee"
I20260808 01:47:58.802074 30614 raft_consensus.cc:3060] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.807411 30991 leader_election.cc:290] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Requested vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.807945 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a3e4affdb54dbeab220185fe1b31d4" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
I20260808 01:47:58.808807 30751 raft_consensus.cc:3060] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.815090 30753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cdada0d44314282a8b07ea0fa3d7774" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6"
W20260808 01:47:58.815994 30800 tablet.cc:2406] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:47:58.816169 30753 raft_consensus.cc:3060] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:58.824642 30987 leader_election.cc:290] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Requested vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:47:58.830557 30751 raft_consensus.cc:2468] T 02a3e4affdb54dbeab220185fe1b31d4 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.835853 30614 raft_consensus.cc:2468] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.837268 30826 leader_election.cc:304] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.838414 30987 raft_consensus.cc:2804] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:58.839057 30987 raft_consensus.cc:697] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Becoming Leader. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Running, Role: LEADER
I20260808 01:47:58.840189 30987 consensus_queue.cc:237] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:47:58.832412 30753 raft_consensus.cc:2468] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.846340 30826 leader_election.cc:304] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters:
I20260808 01:47:58.847409 30991 raft_consensus.cc:2804] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:58.847895 30991 raft_consensus.cc:697] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Becoming Leader. State: Replica: 854c7b716e7c4107b4bee282d113f670, State: Running, Role: LEADER
I20260808 01:47:58.848703 30991 consensus_queue.cc:237] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:47:58.851619 30463 catalog_manager.cc:5697] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854c7b716e7c4107b4bee282d113f670 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:47:58.825736 30613 raft_consensus.cc:2468] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854c7b716e7c4107b4bee282d113f670 in term 1.
I20260808 01:47:58.869273 30463 catalog_manager.cc:5697] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854c7b716e7c4107b4bee282d113f670 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:47:58.870980 30463 catalog_manager.cc:5697] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854c7b716e7c4107b4bee282d113f670 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } }
I20260808 01:47:59.090282 30991 consensus_queue.cc:1048] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.105741 30991 consensus_queue.cc:1048] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.168331 30994 consensus_queue.cc:1048] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.169073 30991 consensus_queue.cc:1048] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.192291 31005 consensus_queue.cc:1048] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.208410 30990 consensus_queue.cc:1048] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.278100 30987 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.293371 30990 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:59.321043 30990 consensus_queue.cc:1048] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:59.336453 30990 consensus_queue.cc:1048] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:00.433386 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 30525
I20260808 01:48:00.473939 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:42159
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:41709
--webserver_port=41087
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:48:00.475301 30826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111)
W20260808 01:48:00.485157 30826 consensus_peers.cc:597] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:00.623256 30826 consensus_peers.cc:597] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:00.699535 30826 consensus_peers.cc:597] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:00.718303 30826 consensus_peers.cc:597] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:00.873404 31039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:00.874118 31039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:00.875061 31039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:00.912034 31039 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:00.912420 31039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:00.913472 31039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:00.954288 31039 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:42159
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=41087
--tserver_master_addrs=127.29.180.126:41709
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:00.955749 31039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:48:00.956710 31039 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:48:00.956869 31039 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:48:00.958091 31039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:00.972240 31046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:01.555546 31040 raft_consensus.cc:493] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25643cac26834dbebc98a8630e53e9ee)
I20260808 01:48:01.556370 31040 raft_consensus.cc:515] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:01.563954 31040 leader_election.cc:290] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:48:01.587051 30751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "854c7b716e7c4107b4bee282d113f670" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" is_pre_election: true
W20260808 01:48:01.598599 30826 leader_election.cc:336] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111)
I20260808 01:48:01.599480 30826 leader_election.cc:304] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670; no voters: 25643cac26834dbebc98a8630e53e9ee, 8f9df0b9b1d04bdab9b2db21bfe249c6
I20260808 01:48:01.601037 31040 raft_consensus.cc:2749] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:48:01.700284 31053 raft_consensus.cc:493] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25643cac26834dbebc98a8630e53e9ee)
I20260808 01:48:01.701241 31053 raft_consensus.cc:515] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
W20260808 01:48:01.717285 30688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111)
I20260808 01:48:01.724936 31053 leader_election.cc:290] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
W20260808 01:48:01.739801 30688 leader_election.cc:336] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111)
I20260808 01:48:01.740020 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670" is_pre_election: true
I20260808 01:48:01.740870 30890 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9df0b9b1d04bdab9b2db21bfe249c6 in term 1.
I20260808 01:48:01.742852 30686 leader_election.cc:304] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters: 25643cac26834dbebc98a8630e53e9ee
I20260808 01:48:01.744240 31053 raft_consensus.cc:2804] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:48:01.744707 31053 raft_consensus.cc:493] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25643cac26834dbebc98a8630e53e9ee)
I20260808 01:48:01.745172 31053 raft_consensus.cc:3060] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:01.755420 31053 raft_consensus.cc:515] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
W20260808 01:48:01.764999 30688 leader_election.cc:336] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111)
I20260808 01:48:01.768296 30890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f57cdc681004e14b9db4f1fa24628e3" candidate_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "854c7b716e7c4107b4bee282d113f670"
I20260808 01:48:01.769091 30890 raft_consensus.cc:3060] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:01.787379 30890 raft_consensus.cc:2468] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9df0b9b1d04bdab9b2db21bfe249c6 in term 2.
I20260808 01:48:01.797796 31053 leader_election.cc:290] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 election: Requested vote from peers 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159), 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:48:01.800263 30686 leader_election.cc:304] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 854c7b716e7c4107b4bee282d113f670, 8f9df0b9b1d04bdab9b2db21bfe249c6; no voters: 25643cac26834dbebc98a8630e53e9ee
I20260808 01:48:01.801990 31053 raft_consensus.cc:2804] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:01.805334 31053 raft_consensus.cc:697] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 2 LEADER]: Becoming Leader. State: Replica: 8f9df0b9b1d04bdab9b2db21bfe249c6, State: Running, Role: LEADER
I20260808 01:48:01.806645 31053 consensus_queue.cc:237] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:01.833808 30464 catalog_manager.cc:5697] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 reported cstate change: term changed from 1 to 2, leader changed from 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) to 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66). New cstate: current_term: 2 leader_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:02.243875 30890 raft_consensus.cc:1275] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Refusing update from remote peer 8f9df0b9b1d04bdab9b2db21bfe249c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:02.247359 31053 consensus_queue.cc:1048] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:48:02.309352 30688 consensus_peers.cc:597] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:02.377683 31045 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 31039
W20260808 01:48:02.709594 31045 kernel_stack_watchdog.cc:198] Thread 31039 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:00.972720 31047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:02.710430 31039 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.738s user 0.581s sys 1.108s
W20260808 01:48:02.710917 31039 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.739s user 0.582s sys 1.109s
W20260808 01:48:02.713129 31049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:02.716890 31048 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1743 milliseconds
I20260808 01:48:02.716890 31039 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:02.718432 31039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:02.720567 31039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:02.722003 31039 hybrid_clock.cc:648] HybridClock initialized: now 1786153682721977 us; error 56 us; skew 500 ppm
I20260808 01:48:02.722900 31039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:02.730307 31039 webserver.cc:533] Webserver started at http://127.29.180.65:41087/ using document root <none> and password file <none>
I20260808 01:48:02.732672 31039 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:02.733089 31039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:48:02.735833 31039 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:48:02.740739 31039 env_posix.cc:431] I/O error, context: INJECTED FAILURE
W20260808 01:48:02.753695 31039 data_dirs.cc:323] instance /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data/block_manager_instance has unknown UUID 92b1d27fae1c43e6b423ee829fef3d40
I20260808 01:48:02.754951 31039 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.010s sys 0.005s
I20260808 01:48:02.757727 31039 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.004s
I20260808 01:48:02.758004 31039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "25643cac26834dbebc98a8630e53e9ee"
format_stamp: "Formatted at 2026-08-08 01:47:53 on dist-test-slave-p3t5"
I20260808 01:48:02.760087 31039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:02.830408 31039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:02.832069 31039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:02.832581 31039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:02.835539 31039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260808 01:48:02.841406 30826 consensus_peers.cc:597] T b15eefc2a48141d3a0700453ac4bbd6d P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:02.843777 31082 ts_tablet_manager.cc:548] Loading tablet metadata (0/5 complete)
I20260808 01:48:02.866583 31039 ts_tablet_manager.cc:585] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260808 01:48:02.866878 31039 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.025s user 0.007s sys 0.008s
I20260808 01:48:02.867180 31039 ts_tablet_manager.cc:600] Registering tablets (0/5 complete)
I20260808 01:48:02.876312 31085 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:48:02.882200 31086 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:48:02.888670 31084 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:48:02.896553 31082 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
E20260808 01:48:02.897249 31082 ts_tablet_manager.cc:1425] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260808 01:48:02.897553 31082 ts_tablet_manager.cc:1403] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 01:48:02.897871 31082 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:02.898204 31082 raft_consensus.cc:2243] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:02.901849 31082 raft_consensus.cc:2272] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:02.903486 31082 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
E20260808 01:48:02.904071 31082 ts_tablet_manager.cc:1425] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260808 01:48:02.904402 31082 ts_tablet_manager.cc:1403] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 01:48:02.904699 31082 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:02.904973 31082 raft_consensus.cc:2243] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:02.905318 31082 raft_consensus.cc:2272] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:02.905887 31039 ts_tablet_manager.cc:616] Registered 5 tablets
I20260808 01:48:02.906164 31039 ts_tablet_manager.cc:595] Time spent register tablets: real 0.039s user 0.025s sys 0.009s
I20260808 01:48:02.977154 31086 log.cc:826] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:02.996521 31085 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:02.998301 31085 tablet_bootstrap.cc:492] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap complete.
I20260808 01:48:03.000113 31086 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:03.001739 31086 tablet_bootstrap.cc:492] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Bootstrap complete.
I20260808 01:48:03.001878 31085 ts_tablet_manager.cc:1403] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.126s user 0.049s sys 0.041s
I20260808 01:48:03.003820 31086 ts_tablet_manager.cc:1403] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.122s user 0.031s sys 0.057s
I20260808 01:48:03.015554 31084 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:03.016965 31084 tablet_bootstrap.cc:492] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap complete.
I20260808 01:48:03.018853 31084 ts_tablet_manager.cc:1403] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.131s user 0.049s sys 0.037s
I20260808 01:48:03.039891 31085 raft_consensus.cc:359] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.039899 31084 raft_consensus.cc:359] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:48:03.041684 31084 raft_consensus.cc:740] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:48:03.042158 31086 raft_consensus.cc:359] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.043215 31086 raft_consensus.cc:740] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:48:03.043380 31085 raft_consensus.cc:740] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:48:03.043036 31084 consensus_queue.cc:260] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } }
I20260808 01:48:03.044236 31085 consensus_queue.cc:260] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.044772 31086 consensus_queue.cc:260] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.053912 31086 ts_tablet_manager.cc:1434] T ecb7b0c5d66e42c4bbc3c3b6c9e3e18d P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.050s user 0.014s sys 0.012s
I20260808 01:48:03.056957 31085 ts_tablet_manager.cc:1434] T b15eefc2a48141d3a0700453ac4bbd6d P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.055s user 0.023s sys 0.013s
I20260808 01:48:03.057435 31084 ts_tablet_manager.cc:1434] T 02a3e4affdb54dbeab220185fe1b31d4 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.038s user 0.006s sys 0.013s
I20260808 01:48:03.151924 31039 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:42159
I20260808 01:48:03.152448 31204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:42159 every 8 connection(s)
I20260808 01:48:03.155006 31039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:48:03.165073 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31039
I20260808 01:48:03.181592 31205 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41709
I20260808 01:48:03.182169 31205 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:03.183285 31205 heartbeater.cc:507] Master 127.29.180.126:41709 requested a full tablet report, sending...
I20260808 01:48:03.189370 30464 ts_manager.cc:194] Re-registered known tserver with Master: 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159)
W20260808 01:48:03.192283 30464 catalog_manager.cc:5610] Tablet 7cdada0d44314282a8b07ea0fa3d7774 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) has failed on TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): IO error: some tablet data is in a failed directory
W20260808 01:48:03.192723 30464 catalog_manager.cc:5610] Tablet 6f57cdc681004e14b9db4f1fa24628e3 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) has failed on TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): IO error: some tablet data is in a failed directory
I20260808 01:48:03.195550 30464 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:37975
I20260808 01:48:03.238596 31205 heartbeater.cc:499] Master 127.29.180.126:41709 was elected leader, sending a full tablet report...
W20260808 01:48:03.245078 30464 catalog_manager.cc:5610] Tablet 7cdada0d44314282a8b07ea0fa3d7774 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) has failed on TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): IO error: some tablet data is in a failed directory
W20260808 01:48:03.245661 30464 catalog_manager.cc:5610] Tablet 6f57cdc681004e14b9db4f1fa24628e3 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) has failed on TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): IO error: some tablet data is in a failed directory
W20260808 01:48:03.281891 30826 consensus_peers.cc:597] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:03.283021 31052 raft_consensus.cc:1012] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Attempting to remove follower 25643cac26834dbebc98a8630e53e9ee from the Raft config. Reason: The tablet replica hosted on peer 25643cac26834dbebc98a8630e53e9ee has failed
I20260808 01:48:03.284996 31052 consensus_queue.cc:237] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.290068 30751 raft_consensus.cc:1275] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Refusing update from remote peer 854c7b716e7c4107b4bee282d113f670: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:03.291600 31217 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:03.299741 31052 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.301762 30751 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.313854 30464 catalog_manager.cc:5697] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: config changed from index -1 to 2, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.336131 30889 consensus_queue.cc:237] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
I20260808 01:48:03.348507 30751 raft_consensus.cc:1275] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Refusing update from remote peer 854c7b716e7c4107b4bee282d113f670: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:48:03.348526 30826 consensus_peers.cc:597] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:03.350674 31217 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:03.352463 31138 tablet_service.cc:1558] Processing DeleteTablet for tablet 7cdada0d44314282a8b07ea0fa3d7774 with delete_type TABLET_DATA_TOMBSTONED (TS 25643cac26834dbebc98a8630e53e9ee not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:53500
I20260808 01:48:03.359543 31225 ts_tablet_manager.cc:1916] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:03.362102 31052 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.366727 30751 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.378922 30449 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7cdada0d44314282a8b07ea0fa3d7774 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:03.397282 30464 catalog_manager.cc:5697] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 reported cstate change: config changed from index 2 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.411405 31225 ts_tablet_manager.cc:1929] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:03.412048 31225 log.cc:1199] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/7cdada0d44314282a8b07ea0fa3d7774
I20260808 01:48:03.418599 31226 raft_consensus.cc:1012] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Attempting to remove follower 25643cac26834dbebc98a8630e53e9ee from the Raft config. Reason: The tablet replica hosted on peer 25643cac26834dbebc98a8630e53e9ee has failed
I20260808 01:48:03.418521 30451 catalog_manager.cc:5028] TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): tablet 7cdada0d44314282a8b07ea0fa3d7774 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) successfully deleted
I20260808 01:48:03.422573 31226 consensus_queue.cc:237] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.431288 30889 raft_consensus.cc:1275] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Refusing update from remote peer 8f9df0b9b1d04bdab9b2db21bfe249c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:48:03.432762 31226 consensus_queue.cc:1048] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:03.440133 31058 raft_consensus.cc:2955] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.441682 30889 raft_consensus.cc:2955] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.454833 30462 catalog_manager.cc:5697] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: config changed from index -1 to 3, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.472538 31138 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f57cdc681004e14b9db4f1fa24628e3 with delete_type TABLET_DATA_TOMBSTONED (TS 25643cac26834dbebc98a8630e53e9ee not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53500
I20260808 01:48:03.474712 31225 ts_tablet_manager.cc:1916] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:03.477341 30751 consensus_queue.cc:237] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
W20260808 01:48:03.490804 30688 consensus_peers.cc:597] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:03.490895 30889 raft_consensus.cc:1275] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Refusing update from remote peer 8f9df0b9b1d04bdab9b2db21bfe249c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:48:03.493245 31230 consensus_queue.cc:1048] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:03.504333 31225 ts_tablet_manager.cc:1929] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:03.504992 31225 log.cc:1199] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/6f57cdc681004e14b9db4f1fa24628e3
I20260808 01:48:03.505046 31058 raft_consensus.cc:2955] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.508246 30451 catalog_manager.cc:5028] TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): tablet 6f57cdc681004e14b9db4f1fa24628e3 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) successfully deleted
I20260808 01:48:03.516722 30889 raft_consensus.cc:2955] T 6f57cdc681004e14b9db4f1fa24628e3 P 854c7b716e7c4107b4bee282d113f670 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.522404 30451 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6f57cdc681004e14b9db4f1fa24628e3 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:03.532862 30462 catalog_manager.cc:5697] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6 reported cstate change: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New cstate: current_term: 2 leader_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:03.774544 31138 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:03.776789 30868 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:03.795435 30731 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:48:03.827831 31227 raft_consensus.cc:1012] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Attempting to remove follower 25643cac26834dbebc98a8630e53e9ee from the Raft config. Reason: The tablet replica hosted on peer 25643cac26834dbebc98a8630e53e9ee has failed
I20260808 01:48:03.830631 31227 consensus_queue.cc:237] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } }
I20260808 01:48:03.840193 30751 raft_consensus.cc:1275] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Refusing update from remote peer 854c7b716e7c4107b4bee282d113f670: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:03.852139 31052 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:03.891041 31217 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.942996 30751 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.981070 30462 catalog_manager.cc:5697] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: config changed from index 3 to 4, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } }
I20260808 01:48:03.999238 31138 tablet_service.cc:1558] Processing DeleteTablet for tablet 7cdada0d44314282a8b07ea0fa3d7774 with delete_type TABLET_DATA_TOMBSTONED (TS 25643cac26834dbebc98a8630e53e9ee not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53500
I20260808 01:48:04.006443 30451 catalog_manager.cc:5028] TS 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): tablet 7cdada0d44314282a8b07ea0fa3d7774 (table test-workload [id=f3b5c23934b844d29bab1a97643d0bef]) successfully deleted
I20260808 01:48:04.004863 30889 consensus_queue.cc:237] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
W20260808 01:48:04.019769 30826 consensus_peers.cc:597] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:04.040683 30753 raft_consensus.cc:1275] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Refusing update from remote peer 854c7b716e7c4107b4bee282d113f670: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 01:48:04.047056 31227 consensus_queue.cc:1048] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:48:04.061690 31251 ts_tablet_manager.cc:933] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Initiating tablet copy from peer 8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925)
I20260808 01:48:04.074146 31251 tablet_copy_client.cc:286] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:48:04.075095 31251 tablet_copy_client.cc:323] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:45925
I20260808 01:48:04.081061 31217 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:04.084432 30751 raft_consensus.cc:2955] T 7cdada0d44314282a8b07ea0fa3d7774 P 8f9df0b9b1d04bdab9b2db21bfe249c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:04.093348 30449 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7cdada0d44314282a8b07ea0fa3d7774 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:04.099649 30462 catalog_manager.cc:5697] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670 reported cstate change: config changed from index 4 to 5, VOTER 25643cac26834dbebc98a8630e53e9ee (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "854c7b716e7c4107b4bee282d113f670" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } } }
I20260808 01:48:04.118444 30773 tablet_copy_service.cc:140] P 8f9df0b9b1d04bdab9b2db21bfe249c6: Received BeginTabletCopySession request for tablet 6f57cdc681004e14b9db4f1fa24628e3 from peer 25643cac26834dbebc98a8630e53e9ee ({username='slave'} at 127.29.180.65:44147)
I20260808 01:48:04.119107 30773 tablet_copy_service.cc:161] P 8f9df0b9b1d04bdab9b2db21bfe249c6: Beginning new tablet copy session on tablet 6f57cdc681004e14b9db4f1fa24628e3 from peer 25643cac26834dbebc98a8630e53e9ee at {username='slave'} at 127.29.180.65:44147: session id = 25643cac26834dbebc98a8630e53e9ee-6f57cdc681004e14b9db4f1fa24628e3
I20260808 01:48:04.127677 30773 tablet_copy_source_session.cc:215] T 6f57cdc681004e14b9db4f1fa24628e3 P 8f9df0b9b1d04bdab9b2db21bfe249c6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:04.131726 31251 ts_tablet_manager.cc:1916] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:04.149501 31251 ts_tablet_manager.cc:1929] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:04.160598 31251 tablet_copy_client.cc:806] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:04.161439 31251 tablet_copy_client.cc:670] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:04.167649 31251 tablet_copy_client.cc:538] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:04.179071 31251 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:48:04.228839 31251 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:04.230245 31251 tablet_bootstrap.cc:492] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap complete.
I20260808 01:48:04.231103 31251 ts_tablet_manager.cc:1403] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.052s user 0.030s sys 0.014s
I20260808 01:48:04.234581 31251 raft_consensus.cc:359] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
I20260808 01:48:04.235459 31251 raft_consensus.cc:740] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:48:04.236390 31251 consensus_queue.cc:260] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
I20260808 01:48:04.246894 31251 ts_tablet_manager.cc:1434] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.015s user 0.004s sys 0.008s
E20260808 01:48:04.247998 31201 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:04.261953 30773 tablet_copy_service.cc:342] P 8f9df0b9b1d04bdab9b2db21bfe249c6: Request end of tablet copy session 25643cac26834dbebc98a8630e53e9ee-6f57cdc681004e14b9db4f1fa24628e3 received from {username='slave'} at 127.29.180.65:44147
I20260808 01:48:04.262624 30773 tablet_copy_service.cc:434] P 8f9df0b9b1d04bdab9b2db21bfe249c6: ending tablet copy session 25643cac26834dbebc98a8630e53e9ee-6f57cdc681004e14b9db4f1fa24628e3 on tablet 6f57cdc681004e14b9db4f1fa24628e3 with peer 25643cac26834dbebc98a8630e53e9ee
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
492e41c2a19d4115953dbd988a8098e9 | 127.29.180.126:41709 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25643cac26834dbebc98a8630e53e9ee | 127.29.180.65:42159 | HEALTHY | <none> | 0 | 0
854c7b716e7c4107b4bee282d113f670 | 127.29.180.67:37937 | HEALTHY | <none> | 4 | 0
8f9df0b9b1d04bdab9b2db21bfe249c6 | 127.29.180.66:45925 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:42159
env_inject_eio | 1 | hidden | 127.29.180.65:42159
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:42159
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:42159
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:45925
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37937
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:42159
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:45925
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37937
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6f57cdc681004e14b9db4f1fa24628e3 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925): RUNNING [LEADER]
854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937): RUNNING
25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 8f9df0b9b1d04bdab9b2db21bfe249c6
B = 854c7b716e7c4107b4bee282d113f670
C = 25643cac26834dbebc98a8630e53e9ee
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 4 | Yes
B | A* B C | 2 | 4 | Yes
C | [config not available] | | |
Tablet 7cdada0d44314282a8b07ea0fa3d7774 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925): RUNNING
854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937): RUNNING [LEADER]
25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 8f9df0b9b1d04bdab9b2db21bfe249c6
B = 854c7b716e7c4107b4bee282d113f670
C = 25643cac26834dbebc98a8630e53e9ee
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 3 | 0 | 2 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:48:04.358881 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:48:04.548050 31158 raft_consensus.cc:1217] T 6f57cdc681004e14b9db4f1fa24628e3 P 25643cac26834dbebc98a8630e53e9ee [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260808 01:48:04.619027 31251 ts_tablet_manager.cc:933] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Initiating tablet copy from peer 854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937)
I20260808 01:48:04.620370 31251 tablet_copy_client.cc:286] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:48:04.620923 31251 tablet_copy_client.cc:323] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:37937
I20260808 01:48:04.631202 30910 tablet_copy_service.cc:140] P 854c7b716e7c4107b4bee282d113f670: Received BeginTabletCopySession request for tablet 7cdada0d44314282a8b07ea0fa3d7774 from peer 25643cac26834dbebc98a8630e53e9ee ({username='slave'} at 127.29.180.65:51873)
I20260808 01:48:04.631819 30910 tablet_copy_service.cc:161] P 854c7b716e7c4107b4bee282d113f670: Beginning new tablet copy session on tablet 7cdada0d44314282a8b07ea0fa3d7774 from peer 25643cac26834dbebc98a8630e53e9ee at {username='slave'} at 127.29.180.65:51873: session id = 25643cac26834dbebc98a8630e53e9ee-7cdada0d44314282a8b07ea0fa3d7774
I20260808 01:48:04.637147 30910 tablet_copy_source_session.cc:215] T 7cdada0d44314282a8b07ea0fa3d7774 P 854c7b716e7c4107b4bee282d113f670: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:04.639801 31251 ts_tablet_manager.cc:1916] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:04.651787 31251 ts_tablet_manager.cc:1929] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:04.658277 31251 tablet_copy_client.cc:806] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:04.658897 31251 tablet_copy_client.cc:670] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:04.663134 31251 tablet_copy_client.cc:538] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:04.670517 31251 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap starting.
I20260808 01:48:04.703013 31251 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:04.704067 31251 tablet_bootstrap.cc:492] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Bootstrap complete.
I20260808 01:48:04.704739 31251 ts_tablet_manager.cc:1403] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.005s
I20260808 01:48:04.707134 31251 raft_consensus.cc:359] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
I20260808 01:48:04.707630 31251 raft_consensus.cc:740] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25643cac26834dbebc98a8630e53e9ee, State: Initialized, Role: FOLLOWER
I20260808 01:48:04.708141 31251 consensus_queue.cc:260] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f9df0b9b1d04bdab9b2db21bfe249c6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 45925 } } peers { permanent_uuid: "854c7b716e7c4107b4bee282d113f670" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37937 } } peers { permanent_uuid: "25643cac26834dbebc98a8630e53e9ee" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42159 } }
I20260808 01:48:04.709898 31251 ts_tablet_manager.cc:1434] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260808 01:48:04.711934 30910 tablet_copy_service.cc:342] P 854c7b716e7c4107b4bee282d113f670: Request end of tablet copy session 25643cac26834dbebc98a8630e53e9ee-7cdada0d44314282a8b07ea0fa3d7774 received from {username='slave'} at 127.29.180.65:51873
I20260808 01:48:04.712407 30910 tablet_copy_service.cc:434] P 854c7b716e7c4107b4bee282d113f670: ending tablet copy session 25643cac26834dbebc98a8630e53e9ee-7cdada0d44314282a8b07ea0fa3d7774 on tablet 7cdada0d44314282a8b07ea0fa3d7774 with peer 25643cac26834dbebc98a8630e53e9ee
I20260808 01:48:04.842693 31138 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:04.856850 30868 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:04.885973 30731 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
492e41c2a19d4115953dbd988a8098e9 | 127.29.180.126:41709 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25643cac26834dbebc98a8630e53e9ee | 127.29.180.65:42159 | HEALTHY | <none> | 0 | 0
854c7b716e7c4107b4bee282d113f670 | 127.29.180.67:37937 | HEALTHY | <none> | 4 | 0
8f9df0b9b1d04bdab9b2db21bfe249c6 | 127.29.180.66:45925 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:42159
env_inject_eio | 1 | hidden | 127.29.180.65:42159
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:42159
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:42159
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:45925
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37937
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:42159
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:45925
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37937
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 7cdada0d44314282a8b07ea0fa3d7774 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
8f9df0b9b1d04bdab9b2db21bfe249c6 (127.29.180.66:45925): RUNNING
854c7b716e7c4107b4bee282d113f670 (127.29.180.67:37937): RUNNING [LEADER]
25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): RUNNING
All reported replicas are:
A = 8f9df0b9b1d04bdab9b2db21bfe249c6
B = 854c7b716e7c4107b4bee282d113f670
C = 25643cac26834dbebc98a8630e53e9ee
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 5 | Yes
B | A B* C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 4 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:48:05.146386 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:48:05.241070 31158 raft_consensus.cc:1217] T 7cdada0d44314282a8b07ea0fa3d7774 P 25643cac26834dbebc98a8630e53e9ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260808 01:48:05.672127 31138 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:05.682027 30868 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:05.682067 30731 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
492e41c2a19d4115953dbd988a8098e9 | 127.29.180.126:41709 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25643cac26834dbebc98a8630e53e9ee | 127.29.180.65:42159 | HEALTHY | <none> | 0 | 0
854c7b716e7c4107b4bee282d113f670 | 127.29.180.67:37937 | HEALTHY | <none> | 4 | 0
8f9df0b9b1d04bdab9b2db21bfe249c6 | 127.29.180.66:45925 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:42159
env_inject_eio | 1 | hidden | 127.29.180.65:42159
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:42159
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:42159
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:45925
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37937
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:42159
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:45925
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37937
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42605 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:48:05.930186 30417 log_verifier.cc:126] Checking tablet 02a3e4affdb54dbeab220185fe1b31d4
I20260808 01:48:06.051899 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 02a3e4affdb54dbeab220185fe1b31d4
I20260808 01:48:06.052376 30417 log_verifier.cc:126] Checking tablet 6f57cdc681004e14b9db4f1fa24628e3
I20260808 01:48:06.169041 30417 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 6f57cdc681004e14b9db4f1fa24628e3
I20260808 01:48:06.169348 30417 log_verifier.cc:126] Checking tablet 7cdada0d44314282a8b07ea0fa3d7774
I20260808 01:48:06.247061 30417 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7cdada0d44314282a8b07ea0fa3d7774
I20260808 01:48:06.247349 30417 log_verifier.cc:126] Checking tablet b15eefc2a48141d3a0700453ac4bbd6d
I20260808 01:48:06.310722 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b15eefc2a48141d3a0700453ac4bbd6d
I20260808 01:48:06.310945 30417 log_verifier.cc:126] Checking tablet ecb7b0c5d66e42c4bbc3c3b6c9e3e18d
I20260808 01:48:06.422147 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ecb7b0c5d66e42c4bbc3c3b6c9e3e18d
I20260808 01:48:07.013439 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31039
I20260808 01:48:07.057929 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 30665
W20260808 01:48:07.070853 30826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260808 01:48:07.074911 30826 consensus_peers.cc:597] T 02a3e4affdb54dbeab220185fe1b31d4 P 854c7b716e7c4107b4bee282d113f670 -> Peer 25643cac26834dbebc98a8630e53e9ee (127.29.180.65:42159): Couldn't send request to peer 25643cac26834dbebc98a8630e53e9ee. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:07.117758 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 30803
I20260808 01:48:07.159143 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 30427
2026-08-08T01:48:07Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0 (18444 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1
2026-08-08T01:48:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:48:07Z Disabled control of system clock
I20260808 01:48:07.291839 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41127
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:41127
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:48:07.621191 31341 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:07.621925 31341 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:07.622423 31341 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:07.657021 31341 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:48:07.657429 31341 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:07.657817 31341 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:48:07.658110 31341 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:48:07.696621 31341 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:41127
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:07.698167 31341 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:07.699926 31341 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:07.711953 31347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:09.116613 31346 debug-util.cc:398] Leaking SignalData structure 0x7b08000290e0 after lost signal to thread 31341
W20260808 01:48:09.658092 31346 kernel_stack_watchdog.cc:198] Thread 31341 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:09.659349 31341 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.947s user 0.651s sys 1.292s
W20260808 01:48:07.712399 31348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:09.659780 31341 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.948s user 0.651s sys 1.292s
W20260808 01:48:09.662204 31350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:09.665776 31349 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1947 milliseconds
I20260808 01:48:09.665779 31341 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:09.667472 31341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:09.670315 31341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:09.671737 31341 hybrid_clock.cc:648] HybridClock initialized: now 1786153689671640 us; error 134 us; skew 500 ppm
I20260808 01:48:09.672727 31341 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:09.680315 31341 webserver.cc:533] Webserver started at http://127.29.180.126:38447/ using document root <none> and password file <none>
I20260808 01:48:09.682210 31341 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:09.682538 31341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:09.683332 31341 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:09.688555 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.689991 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.691064 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.692180 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.693248 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.694303 31341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.717433 31341 fs_manager.cc:696] Time spent creating directory manager: real 0.022s user 0.017s sys 0.005s
I20260808 01:48:09.722309 31341 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:09.722617 31341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "df3e745e81ab4066b37ecdf9252eaf54"
format_stamp: "Formatted at 2026-08-08 01:48:09 on dist-test-slave-p3t5"
I20260808 01:48:09.723057 31341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:09.776024 31341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:09.778452 31341 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:09.779191 31341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:09.856303 31341 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:41127
I20260808 01:48:09.856390 31414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:41127 every 8 connection(s)
I20260808 01:48:09.859234 31341 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:48:09.864423 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31341
I20260808 01:48:09.865209 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:48:09.892010 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54: Bootstrap starting.
I20260808 01:48:09.899221 31415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:09.901377 31415 log.cc:826] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:09.906924 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54: No bootstrap required, opened a new log
I20260808 01:48:09.927103 31415 raft_consensus.cc:359] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } }
I20260808 01:48:09.927814 31415 raft_consensus.cc:385] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:09.928025 31415 raft_consensus.cc:740] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3e745e81ab4066b37ecdf9252eaf54, State: Initialized, Role: FOLLOWER
I20260808 01:48:09.928814 31415 consensus_queue.cc:260] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } }
I20260808 01:48:09.929327 31415 raft_consensus.cc:399] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:09.929584 31415 raft_consensus.cc:493] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:09.929908 31415 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:09.934059 31415 raft_consensus.cc:515] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } }
I20260808 01:48:09.934758 31415 leader_election.cc:304] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df3e745e81ab4066b37ecdf9252eaf54; no voters:
I20260808 01:48:09.936720 31415 leader_election.cc:290] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:48:09.937238 31420 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:09.940858 31420 raft_consensus.cc:697] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [term 1 LEADER]: Becoming Leader. State: Replica: df3e745e81ab4066b37ecdf9252eaf54, State: Running, Role: LEADER
I20260808 01:48:09.941737 31415 sys_catalog.cc:565] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:09.941761 31420 consensus_queue.cc:237] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } }
I20260808 01:48:09.952436 31422 sys_catalog.cc:455] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df3e745e81ab4066b37ecdf9252eaf54. Latest consensus state: current_term: 1 leader_uuid: "df3e745e81ab4066b37ecdf9252eaf54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } } }
I20260808 01:48:09.953122 31422 sys_catalog.cc:458] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:09.954859 31421 sys_catalog.cc:455] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df3e745e81ab4066b37ecdf9252eaf54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3e745e81ab4066b37ecdf9252eaf54" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41127 } } }
I20260808 01:48:09.955919 31421 sys_catalog.cc:458] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54 [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:09.957929 31428 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:09.969151 31428 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:09.988176 31428 catalog_manager.cc:1383] Generated new cluster ID: ee7bd5fae1d3462d8dcba357594ceffd
I20260808 01:48:09.988547 31428 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:10.025985 31428 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:10.027710 31428 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:10.042889 31428 catalog_manager.cc:6070] T 00000000000000000000000000000000 P df3e745e81ab4066b37ecdf9252eaf54: Generated new TSK 0
I20260808 01:48:10.044188 31428 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:10.066578 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:48:10.402447 31440 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:10.403039 31440 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:10.403589 31440 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:10.439786 31440 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:10.440757 31440 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:10.481799 31440 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:10.483287 31440 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:10.485116 31440 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:10.499586 31446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:10.501248 31447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:12.072180 31440 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.572s user 0.499s sys 0.991s
W20260808 01:48:11.905138 31445 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 31440
W20260808 01:48:12.073379 31440 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.574s user 0.499s sys 0.992s
W20260808 01:48:12.075376 31449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:12.078022 31448 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1571 milliseconds
I20260808 01:48:12.078076 31440 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:12.079710 31440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:12.082525 31440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:12.084060 31440 hybrid_clock.cc:648] HybridClock initialized: now 1786153692084004 us; error 54 us; skew 500 ppm
I20260808 01:48:12.085229 31440 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:12.093411 31440 webserver.cc:533] Webserver started at http://127.29.180.65:33469/ using document root <none> and password file <none>
I20260808 01:48:12.095685 31440 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:12.096065 31440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:12.096997 31440 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:12.103787 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.105504 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.106884 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.108150 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.109390 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.110705 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.136116 31440 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.011s sys 0.013s
I20260808 01:48:12.140715 31440 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:12.140965 31440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:12.141410 31440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:12.196822 31440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:12.198455 31440 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:12.198953 31440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:12.202265 31440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:12.207314 31440 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:12.207549 31440 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:12.207908 31440 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:12.208079 31440 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:12.399149 31440 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:42955
I20260808 01:48:12.399348 31573 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:42955 every 8 connection(s)
I20260808 01:48:12.402081 31440 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:48:12.412333 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31440
I20260808 01:48:12.412807 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:48:12.419831 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:12.431874 31574 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41127
I20260808 01:48:12.432459 31574 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:12.434006 31574 heartbeater.cc:507] Master 127.29.180.126:41127 requested a full tablet report, sending...
I20260808 01:48:12.437845 31379 ts_manager.cc:194] Registered new tserver with Master: dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:12.441030 31379 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:56059
W20260808 01:48:12.763638 31579 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:12.764242 31579 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:12.764770 31579 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:12.798561 31579 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:12.799486 31579 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:48:12.836835 31579 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:12.838220 31579 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:12.839871 31579 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:12.855592 31585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:13.447022 31574 heartbeater.cc:499] Master 127.29.180.126:41127 was elected leader, sending a full tablet report...
W20260808 01:48:14.257995 31584 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 31579
W20260808 01:48:15.016146 31579 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.162s user 0.739s sys 1.412s
W20260808 01:48:12.857034 31586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:15.017791 31579 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.164s user 0.741s sys 1.412s
W20260808 01:48:15.023496 31588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:15.028393 31587 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2161 milliseconds
I20260808 01:48:15.028475 31579 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:15.031126 31579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:15.036758 31579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:15.040963 31579 hybrid_clock.cc:648] HybridClock initialized: now 1786153695040894 us; error 172 us; skew 500 ppm
I20260808 01:48:15.042829 31579 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:15.057971 31579 webserver.cc:533] Webserver started at http://127.29.180.66:34103/ using document root <none> and password file <none>
I20260808 01:48:15.062014 31579 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:15.062670 31579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:15.064060 31579 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:15.076766 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.080361 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.081996 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.083906 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.086272 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.088642 31579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.140044 31579 fs_manager.cc:696] Time spent creating directory manager: real 0.050s user 0.029s sys 0.018s
I20260808 01:48:15.150050 31579 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:15.150547 31579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "b3f387bcc2a042bea411330031f4cb3c"
format_stamp: "Formatted at 2026-08-08 01:48:15 on dist-test-slave-p3t5"
I20260808 01:48:15.151283 31579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:15.263777 31579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:15.266176 31579 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:15.266918 31579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:15.270881 31579 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:15.280092 31579 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:15.280695 31579 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260808 01:48:15.281433 31579 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:15.281996 31579 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.003s sys 0.000s
I20260808 01:48:15.552935 31579 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:42495
I20260808 01:48:15.553198 31711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:42495 every 8 connection(s)
I20260808 01:48:15.558871 31579 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:48:15.564107 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31579
I20260808 01:48:15.565271 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:48:15.581724 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:15.616051 31712 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41127
I20260808 01:48:15.617195 31712 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:15.619540 31712 heartbeater.cc:507] Master 127.29.180.126:41127 requested a full tablet report, sending...
I20260808 01:48:15.625682 31379 ts_manager.cc:194] Registered new tserver with Master: b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495)
I20260808 01:48:15.629299 31379 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:56623
W20260808 01:48:16.195775 31717 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:16.196813 31717 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:16.197688 31717 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:16.254134 31717 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:16.255505 31717 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:48:16.319761 31717 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:16.322177 31717 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:16.325264 31717 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:16.355676 31723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:16.638283 31712 heartbeater.cc:499] Master 127.29.180.126:41127 was elected leader, sending a full tablet report...
W20260808 01:48:16.356457 31724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:18.272917 31726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:18.276360 31725 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1919 milliseconds
I20260808 01:48:18.276487 31717 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:18.278124 31717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:18.280666 31717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:18.282143 31717 hybrid_clock.cc:648] HybridClock initialized: now 1786153698282076 us; error 69 us; skew 500 ppm
I20260808 01:48:18.283159 31717 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:18.292019 31717 webserver.cc:533] Webserver started at http://127.29.180.67:46115/ using document root <none> and password file <none>
I20260808 01:48:18.293700 31717 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:18.294024 31717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:18.294757 31717 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:18.299778 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.301067 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.302043 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.303085 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.304479 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.305843 31717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.329860 31717 fs_manager.cc:696] Time spent creating directory manager: real 0.023s user 0.017s sys 0.006s
I20260808 01:48:18.334753 31717 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:18.335064 31717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "123f7f87d8874aa2adab8d5e011cf479"
format_stamp: "Formatted at 2026-08-08 01:48:18 on dist-test-slave-p3t5"
I20260808 01:48:18.335490 31717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:18.392970 31717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:18.394797 31717 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:18.395403 31717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:18.399062 31717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:18.404367 31717 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:18.404630 31717 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:18.404927 31717 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:18.405135 31717 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:18.589053 31717 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:33353
I20260808 01:48:18.589169 31849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:33353 every 8 connection(s)
I20260808 01:48:18.591975 31717 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:48:18.596419 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31717
I20260808 01:48:18.596989 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:48:18.605959 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.68:0
--local_ip_for_outbound_sockets=127.29.180.68
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:18.617947 31850 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41127
I20260808 01:48:18.618456 31850 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:18.619611 31850 heartbeater.cc:507] Master 127.29.180.126:41127 requested a full tablet report, sending...
I20260808 01:48:18.622467 31379 ts_manager.cc:194] Registered new tserver with Master: 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353)
I20260808 01:48:18.624130 31379 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:50627
W20260808 01:48:18.948621 31854 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:18.949226 31854 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:18.949813 31854 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:18.985488 31854 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:18.986454 31854 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.68
I20260808 01:48:19.025568 31854 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41127
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:19.027143 31854 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:19.028918 31854 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:19.043510 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:19.627662 31850 heartbeater.cc:499] Master 127.29.180.126:41127 was elected leader, sending a full tablet report...
W20260808 01:48:19.045425 31864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:20.282555 31866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:20.284766 31865 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1234 milliseconds
I20260808 01:48:20.284881 31854 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:20.286365 31854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:20.289736 31854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:20.291266 31854 hybrid_clock.cc:648] HybridClock initialized: now 1786153700291219 us; error 79 us; skew 500 ppm
I20260808 01:48:20.292212 31854 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:20.299520 31854 webserver.cc:533] Webserver started at http://127.29.180.68:35729/ using document root <none> and password file <none>
I20260808 01:48:20.301143 31854 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:20.301450 31854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:20.302264 31854 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:20.307221 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.308468 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-1/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.309419 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-2/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.310379 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-3/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.311472 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-4/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.312399 31854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance:
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.335975 31854 fs_manager.cc:696] Time spent creating directory manager: real 0.023s user 0.010s sys 0.014s
I20260808 01:48:20.340907 31854 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:20.341179 31854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal
uuid: "805e196f2b1e48e2a598169ff776c113"
format_stamp: "Formatted at 2026-08-08 01:48:20 on dist-test-slave-p3t5"
I20260808 01:48:20.341774 31854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:20.403405 31854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:20.405215 31854 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:20.405900 31854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:20.409571 31854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:20.414882 31854 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:20.415149 31854 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:20.415426 31854 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:20.415628 31854 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:20.634366 31854 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.68:45233
I20260808 01:48:20.634788 31991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.68:45233 every 8 connection(s)
I20260808 01:48:20.637414 31854 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
I20260808 01:48:20.645987 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 31854
I20260808 01:48:20.647977 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance
I20260808 01:48:20.686246 31992 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41127
I20260808 01:48:20.686800 31992 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:20.687989 31992 heartbeater.cc:507] Master 127.29.180.126:41127 requested a full tablet report, sending...
I20260808 01:48:20.690564 31379 ts_manager.cc:194] Registered new tserver with Master: 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233)
I20260808 01:48:20.692250 31379 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.68:50057
I20260808 01:48:20.695873 30417 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20260808 01:48:20.790154 31379 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57000:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:48:20.902930 31925 tablet_service.cc:1511] Processing CreateTablet for tablet a6b4daa349484e79875550e1f976021d (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:20.904733 31644 tablet_service.cc:1511] Processing CreateTablet for tablet f165e239f6fd412d8ba2c66079f53543 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:20.906139 31645 tablet_service.cc:1511] Processing CreateTablet for tablet 14ed40bb7c6c4b5fa28e64dfcacf50f8 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:20.906477 31924 tablet_service.cc:1511] Processing CreateTablet for tablet 474ef75c3b7c4b13af5047a850a04d07 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:20.910132 31923 tablet_service.cc:1511] Processing CreateTablet for tablet f165e239f6fd412d8ba2c66079f53543 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:20.911044 31646 tablet_service.cc:1511] Processing CreateTablet for tablet a6b4daa349484e79875550e1f976021d (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:20.914842 31643 tablet_service.cc:1511] Processing CreateTablet for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:20.946019 31504 tablet_service.cc:1511] Processing CreateTablet for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:20.948799 31506 tablet_service.cc:1511] Processing CreateTablet for tablet 14ed40bb7c6c4b5fa28e64dfcacf50f8 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:20.956552 31782 tablet_service.cc:1511] Processing CreateTablet for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:20.958695 31507 tablet_service.cc:1511] Processing CreateTablet for tablet a6b4daa349484e79875550e1f976021d (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:20.961499 31783 tablet_service.cc:1511] Processing CreateTablet for tablet f165e239f6fd412d8ba2c66079f53543 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:20.966094 31784 tablet_service.cc:1511] Processing CreateTablet for tablet 474ef75c3b7c4b13af5047a850a04d07 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:20.957113 31785 tablet_service.cc:1511] Processing CreateTablet for tablet 14ed40bb7c6c4b5fa28e64dfcacf50f8 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:20.973984 31505 tablet_service.cc:1511] Processing CreateTablet for tablet 474ef75c3b7c4b13af5047a850a04d07 (DEFAULT_TABLE table=test-workload [id=24b28d797daa4674942b43d523058529]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:21.000949 32019 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: Bootstrap starting.
I20260808 01:48:21.009446 32020 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c: Bootstrap starting.
I20260808 01:48:21.023674 32018 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: Bootstrap starting.
I20260808 01:48:21.034837 32019 tablet_bootstrap.cc:654] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.040184 32019 log.cc:826] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:21.040972 32020 tablet_bootstrap.cc:654] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.051697 32023 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: Bootstrap starting.
I20260808 01:48:21.054641 32021 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c: Bootstrap starting.
I20260808 01:48:21.072378 32026 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113: Bootstrap starting.
I20260808 01:48:21.075688 32022 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: Bootstrap starting.
I20260808 01:48:21.078279 32023 tablet_bootstrap.cc:654] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.091956 32020 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c: No bootstrap required, opened a new log
I20260808 01:48:21.092783 32020 ts_tablet_manager.cc:1403] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.011s
I20260808 01:48:21.100172 32023 log.cc:826] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:21.100731 32027 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Bootstrap starting.
I20260808 01:48:21.114009 32027 tablet_bootstrap.cc:654] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.114514 32026 tablet_bootstrap.cc:654] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.120774 32027 log.cc:826] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:21.127121 32023 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: No bootstrap required, opened a new log
I20260808 01:48:21.127744 32023 ts_tablet_manager.cc:1403] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: Time spent bootstrapping tablet: real 0.077s user 0.015s sys 0.008s
I20260808 01:48:21.131639 32022 tablet_bootstrap.cc:654] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.141734 32026 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113: No bootstrap required, opened a new log
I20260808 01:48:21.142252 32026 ts_tablet_manager.cc:1403] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.006s
I20260808 01:48:21.173990 32019 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: No bootstrap required, opened a new log
I20260808 01:48:21.174121 32023 raft_consensus.cc:359] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.174865 32019 ts_tablet_manager.cc:1403] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: Time spent bootstrapping tablet: real 0.175s user 0.057s sys 0.063s
I20260808 01:48:21.175115 32023 raft_consensus.cc:385] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.175428 32023 raft_consensus.cc:740] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805e196f2b1e48e2a598169ff776c113, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.177469 32018 tablet_bootstrap.cc:654] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.176798 32023 consensus_queue.cc:260] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.180604 32026 raft_consensus.cc:359] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.183673 31992 heartbeater.cc:499] Master 127.29.180.126:41127 was elected leader, sending a full tablet report...
I20260808 01:48:21.179713 32021 tablet_bootstrap.cc:654] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.184551 32023 ts_tablet_manager.cc:1434] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113: Time spent starting tablet: real 0.056s user 0.013s sys 0.012s
I20260808 01:48:21.186270 32022 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: No bootstrap required, opened a new log
I20260808 01:48:21.186937 32034 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:21.192368 32033 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:21.198508 32022 ts_tablet_manager.cc:1403] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: Time spent bootstrapping tablet: real 0.123s user 0.014s sys 0.018s
I20260808 01:48:21.190135 32026 raft_consensus.cc:385] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.201874 32026 raft_consensus.cc:740] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805e196f2b1e48e2a598169ff776c113, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.202924 32026 consensus_queue.cc:260] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.205698 32026 ts_tablet_manager.cc:1434] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113: Time spent starting tablet: real 0.063s user 0.018s sys 0.006s
I20260808 01:48:21.215823 32027 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: No bootstrap required, opened a new log
I20260808 01:48:21.222250 32027 ts_tablet_manager.cc:1403] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Time spent bootstrapping tablet: real 0.122s user 0.022s sys 0.011s
I20260808 01:48:21.229231 32036 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:21.245348 32018 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: No bootstrap required, opened a new log
I20260808 01:48:21.245985 32018 ts_tablet_manager.cc:1403] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: Time spent bootstrapping tablet: real 0.223s user 0.008s sys 0.009s
I20260808 01:48:21.248641 32040 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479: Bootstrap starting.
I20260808 01:48:21.250519 32037 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:21.250947 32022 raft_consensus.cc:359] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.251784 32022 raft_consensus.cc:385] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.252159 32022 raft_consensus.cc:740] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805e196f2b1e48e2a598169ff776c113, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.253062 32022 consensus_queue.cc:260] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.252198 32034 tablet_bootstrap.cc:654] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.258555 32019 raft_consensus.cc:359] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.259379 32019 raft_consensus.cc:385] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.259716 32019 raft_consensus.cc:740] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.261861 32022 ts_tablet_manager.cc:1434] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260808 01:48:21.260818 32019 consensus_queue.cc:260] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.256937 32020 raft_consensus.cc:359] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.262579 32020 raft_consensus.cc:385] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.262974 32020 raft_consensus.cc:740] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.263690 32020 consensus_queue.cc:260] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.264710 32021 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c: No bootstrap required, opened a new log
I20260808 01:48:21.265267 32021 ts_tablet_manager.cc:1403] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c: Time spent bootstrapping tablet: real 0.211s user 0.012s sys 0.005s
I20260808 01:48:21.266803 32035 raft_consensus.cc:493] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.267880 32033 tablet_bootstrap.cc:654] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.267437 32035 raft_consensus.cc:515] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.270646 32039 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: Bootstrap starting.
I20260808 01:48:21.269071 32018 raft_consensus.cc:359] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.271484 32018 raft_consensus.cc:385] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.271843 32018 raft_consensus.cc:740] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.275300 32021 raft_consensus.cc:359] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.276678 32034 log.cc:826] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:21.278924 32041 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: Bootstrap starting.
I20260808 01:48:21.283447 32021 raft_consensus.cc:385] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.293572 32035 leader_election.cc:290] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.298749 32021 raft_consensus.cc:740] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.283851 32020 ts_tablet_manager.cc:1434] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c: Time spent starting tablet: real 0.191s user 0.026s sys 0.015s
I20260808 01:48:21.273007 32018 consensus_queue.cc:260] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.303954 32018 ts_tablet_manager.cc:1434] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: Time spent starting tablet: real 0.058s user 0.006s sys 0.002s
I20260808 01:48:21.299896 32019 ts_tablet_manager.cc:1434] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c: Time spent starting tablet: real 0.125s user 0.031s sys 0.009s
I20260808 01:48:21.306138 32021 consensus_queue.cc:260] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.308601 32021 ts_tablet_manager.cc:1434] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260808 01:48:21.314715 32039 tablet_bootstrap.cc:654] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.315724 32041 tablet_bootstrap.cc:654] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.307307 32036 tablet_bootstrap.cc:654] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.318708 32040 tablet_bootstrap.cc:654] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:21.320689 32037 tablet_bootstrap.cc:654] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Neither blocks nor log segments found. Creating new log.
W20260808 01:48:21.328136 31713 tablet.cc:2406] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:21.328562 31666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "805e196f2b1e48e2a598169ff776c113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c" is_pre_election: true
I20260808 01:48:21.330030 31666 raft_consensus.cc:2468] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 805e196f2b1e48e2a598169ff776c113 in term 0.
I20260808 01:48:21.332930 31881 leader_election.cc:304] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c; no voters:
I20260808 01:48:21.334157 32035 raft_consensus.cc:2804] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:21.334614 32035 raft_consensus.cc:493] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:21.335001 32035 raft_consensus.cc:3060] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.345305 32035 raft_consensus.cc:515] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.349123 31666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "805e196f2b1e48e2a598169ff776c113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c"
I20260808 01:48:21.349793 31666 raft_consensus.cc:3060] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.350221 32035 leader_election.cc:290] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 election: Requested vote from peers b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.351524 32040 tablet_bootstrap.cc:492] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479: No bootstrap required, opened a new log
I20260808 01:48:21.352020 32040 ts_tablet_manager.cc:1403] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.005s
I20260808 01:48:21.355458 32034 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: No bootstrap required, opened a new log
I20260808 01:48:21.356029 32034 ts_tablet_manager.cc:1403] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.170s user 0.014s sys 0.010s
I20260808 01:48:21.357158 31666 raft_consensus.cc:2468] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 805e196f2b1e48e2a598169ff776c113 in term 1.
I20260808 01:48:21.357676 32039 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: No bootstrap required, opened a new log
I20260808 01:48:21.358129 32039 ts_tablet_manager.cc:1403] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.006s
I20260808 01:48:21.358835 31881 leader_election.cc:304] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c; no voters:
I20260808 01:48:21.360040 32035 raft_consensus.cc:2804] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:21.354410 31527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "805e196f2b1e48e2a598169ff776c113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35"
I20260808 01:48:21.350354 31528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "805e196f2b1e48e2a598169ff776c113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35" is_pre_election: true
I20260808 01:48:21.364836 32041 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: No bootstrap required, opened a new log
I20260808 01:48:21.364938 32033 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: No bootstrap required, opened a new log
I20260808 01:48:21.365432 32041 ts_tablet_manager.cc:1403] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.011s
I20260808 01:48:21.368929 32037 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: No bootstrap required, opened a new log
I20260808 01:48:21.369520 32037 ts_tablet_manager.cc:1403] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.006s
W20260808 01:48:21.370287 31878 leader_election.cc:343] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:48:21.372015 31878 leader_election.cc:343] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:21.365403 32033 ts_tablet_manager.cc:1403] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.004s
I20260808 01:48:21.374490 32043 raft_consensus.cc:493] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.375028 32043 raft_consensus.cc:515] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.377094 32035 raft_consensus.cc:697] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [term 1 LEADER]: Becoming Leader. State: Replica: 805e196f2b1e48e2a598169ff776c113, State: Running, Role: LEADER
I20260808 01:48:21.380365 32043 leader_election.cc:290] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353), 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233)
I20260808 01:48:21.380223 32035 consensus_queue.cc:237] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.386955 32041 raft_consensus.cc:359] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.387961 32041 raft_consensus.cc:385] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.388262 32041 raft_consensus.cc:740] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.389308 32041 consensus_queue.cc:260] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.392244 32039 raft_consensus.cc:359] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.393342 32039 raft_consensus.cc:385] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.395241 32039 raft_consensus.cc:740] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.396018 32033 raft_consensus.cc:359] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.394459 32027 raft_consensus.cc:359] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.397961 32027 raft_consensus.cc:385] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.397058 32033 raft_consensus.cc:385] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.398943 32033 raft_consensus.cc:740] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.396538 32039 consensus_queue.cc:260] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.399163 32027 raft_consensus.cc:740] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.400455 32033 consensus_queue.cc:260] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.402050 32027 consensus_queue.cc:260] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
W20260808 01:48:21.403786 31993 tablet.cc:2406] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:21.403825 32037 raft_consensus.cc:359] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.404636 32037 raft_consensus.cc:385] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.404997 32037 raft_consensus.cc:740] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.405824 32037 consensus_queue.cc:260] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.405753 32040 raft_consensus.cc:359] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.410871 32040 raft_consensus.cc:385] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.411381 32036 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: No bootstrap required, opened a new log
I20260808 01:48:21.411926 32036 ts_tablet_manager.cc:1403] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.183s user 0.015s sys 0.006s
I20260808 01:48:21.412421 32040 raft_consensus.cc:740] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.418488 32040 consensus_queue.cc:260] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.422592 32037 ts_tablet_manager.cc:1434] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.053s user 0.017s sys 0.001s
W20260808 01:48:21.424763 31576 tablet.cc:2406] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:21.429827 32036 raft_consensus.cc:359] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.430725 32036 raft_consensus.cc:385] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.431105 32036 raft_consensus.cc:740] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.432000 32027 ts_tablet_manager.cc:1434] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Time spent starting tablet: real 0.209s user 0.034s sys 0.000s
I20260808 01:48:21.431946 32036 consensus_queue.cc:260] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.434203 32040 ts_tablet_manager.cc:1434] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479: Time spent starting tablet: real 0.082s user 0.014s sys 0.000s
I20260808 01:48:21.443807 32041 ts_tablet_manager.cc:1434] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: Time spent starting tablet: real 0.078s user 0.017s sys 0.001s
I20260808 01:48:21.443490 32034 raft_consensus.cc:359] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.446875 32034 raft_consensus.cc:385] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:21.448446 32034 raft_consensus.cc:740] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:21.450863 32034 consensus_queue.cc:260] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.449354 32036 ts_tablet_manager.cc:1434] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.037s user 0.009s sys 0.000s
I20260808 01:48:21.456207 32034 ts_tablet_manager.cc:1434] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.100s user 0.011s sys 0.008s
I20260808 01:48:21.459646 32033 ts_tablet_manager.cc:1434] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.086s user 0.021s sys 0.005s
I20260808 01:48:21.461450 32058 raft_consensus.cc:493] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.463577 31947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113" is_pre_election: true
I20260808 01:48:21.464445 31947 raft_consensus.cc:2468] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 0.
I20260808 01:48:21.465996 32058 raft_consensus.cc:515] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.466610 31600 leader_election.cc:304] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c; no voters:
I20260808 01:48:21.467242 32059 raft_consensus.cc:493] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.467980 32043 raft_consensus.cc:2804] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:21.467758 32059 raft_consensus.cc:515] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.468345 32043 raft_consensus.cc:493] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:21.463912 32039 ts_tablet_manager.cc:1434] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: Time spent starting tablet: real 0.105s user 0.013s sys 0.007s
I20260808 01:48:21.468714 32043 raft_consensus.cc:3060] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.476702 32043 raft_consensus.cc:515] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.480345 32043 leader_election.cc:290] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 election: Requested vote from peers 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353), 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233)
I20260808 01:48:21.480383 32058 leader_election.cc:290] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.482445 31947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113"
I20260808 01:48:21.484426 31947 raft_consensus.cc:3060] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.492643 31947 raft_consensus.cc:2468] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 1.
I20260808 01:48:21.497175 32058 raft_consensus.cc:493] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.497746 32058 raft_consensus.cc:515] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.499774 32058 leader_election.cc:290] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955), b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495)
I20260808 01:48:21.498726 31600 leader_election.cc:304] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c; no voters:
I20260808 01:48:21.501739 32043 raft_consensus.cc:2804] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:21.500592 31378 catalog_manager.cc:5697] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 reported cstate change: term changed from 0 to 1, leader changed from <none> to 805e196f2b1e48e2a598169ff776c113 (127.29.180.68). New cstate: current_term: 1 leader_uuid: "805e196f2b1e48e2a598169ff776c113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:21.502736 32043 raft_consensus.cc:697] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEADER]: Becoming Leader. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Running, Role: LEADER
I20260808 01:48:21.503713 32043 consensus_queue.cc:237] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:21.508389 31805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f7f87d8874aa2adab8d5e011cf479" is_pre_election: true
I20260808 01:48:21.509301 31805 raft_consensus.cc:2468] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 0.
I20260808 01:48:21.510929 31804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f7f87d8874aa2adab8d5e011cf479"
I20260808 01:48:21.511574 31804 raft_consensus.cc:3060] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.521787 31804 raft_consensus.cc:2468] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 1.
I20260808 01:48:21.524063 32060 raft_consensus.cc:493] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.525215 32060 raft_consensus.cc:515] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.535997 31947 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "474ef75c3b7c4b13af5047a850a04d07" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113" is_pre_election: true
I20260808 01:48:21.536629 31947 raft_consensus.cc:2468] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 0.
I20260808 01:48:21.536471 31946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113" is_pre_election: true
I20260808 01:48:21.537530 31946 raft_consensus.cc:2393] T f165e239f6fd412d8ba2c66079f53543 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123f7f87d8874aa2adab8d5e011cf479 in current term 1: Already voted for candidate b3f387bcc2a042bea411330031f4cb3c in this term.
I20260808 01:48:21.545763 32060 leader_election.cc:290] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233), b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495)
I20260808 01:48:21.545758 31377 catalog_manager.cc:5697] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b3f387bcc2a042bea411330031f4cb3c (127.29.180.66). New cstate: current_term: 1 leader_uuid: "b3f387bcc2a042bea411330031f4cb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:21.571611 31528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "474ef75c3b7c4b13af5047a850a04d07" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35" is_pre_election: true
I20260808 01:48:21.572383 31528 raft_consensus.cc:2468] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 0.
I20260808 01:48:21.573851 31527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6c1a2fdd3214f9c8cf5450f73b05e5b" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35" is_pre_election: true
I20260808 01:48:21.574417 31527 raft_consensus.cc:2468] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 0.
I20260808 01:48:21.522497 32059 leader_election.cc:290] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495), 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233)
I20260808 01:48:21.538403 31739 leader_election.cc:304] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, 805e196f2b1e48e2a598169ff776c113; no voters:
W20260808 01:48:21.602223 31739 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.180.68:45233, user_credentials={real_user=slave}} blocked reactor thread for 64075.2us
W20260808 01:48:21.603061 31851 tablet.cc:2406] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:21.603931 32058 raft_consensus.cc:2804] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:21.604410 32058 raft_consensus.cc:493] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:21.604835 32058 raft_consensus.cc:3060] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.616196 32043 raft_consensus.cc:493] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.616709 32043 raft_consensus.cc:515] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.617487 31738 leader_election.cc:304] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, dd45cf41d4134d1e84318ca9aee76f35; no voters:
I20260808 01:48:21.618988 32043 leader_election.cc:290] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.619681 32062 raft_consensus.cc:2804] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:21.620024 32062 raft_consensus.cc:493] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:21.620325 32062 raft_consensus.cc:3060] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.627709 32058 raft_consensus.cc:515] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.635927 31666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f165e239f6fd412d8ba2c66079f53543" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c" is_pre_election: true
I20260808 01:48:21.637660 31804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f7f87d8874aa2adab8d5e011cf479" is_pre_election: true
I20260808 01:48:21.638531 31804 raft_consensus.cc:2468] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 0.
I20260808 01:48:21.639040 31666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6c1a2fdd3214f9c8cf5450f73b05e5b" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c" is_pre_election: true
I20260808 01:48:21.639627 31666 raft_consensus.cc:2468] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 0.
I20260808 01:48:21.640594 31601 leader_election.cc:304] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, b3f387bcc2a042bea411330031f4cb3c; no voters:
I20260808 01:48:21.641803 32043 raft_consensus.cc:2804] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:21.642254 32043 raft_consensus.cc:493] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:21.642627 32043 raft_consensus.cc:3060] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.650718 32062 raft_consensus.cc:515] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.651360 31946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "474ef75c3b7c4b13af5047a850a04d07" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113"
I20260808 01:48:21.652002 31946 raft_consensus.cc:3060] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.652899 32058 leader_election.cc:290] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 election: Requested vote from peers 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.653964 31527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6c1a2fdd3214f9c8cf5450f73b05e5b" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35"
I20260808 01:48:21.654574 31527 raft_consensus.cc:3060] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.657763 32060 raft_consensus.cc:493] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:21.658221 32060 raft_consensus.cc:515] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.660547 32060 leader_election.cc:290] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353), b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495)
I20260808 01:48:21.662580 31741 leader_election.cc:304] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479; no voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c
I20260808 01:48:21.663530 31946 raft_consensus.cc:2468] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 1.
I20260808 01:48:21.664605 31739 leader_election.cc:304] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, 805e196f2b1e48e2a598169ff776c113; no voters:
I20260808 01:48:21.665445 32059 raft_consensus.cc:2749] T f165e239f6fd412d8ba2c66079f53543 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:21.666143 32059 raft_consensus.cc:2804] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:21.666580 32059 raft_consensus.cc:697] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Becoming Leader. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Running, Role: LEADER
I20260808 01:48:21.667625 32059 consensus_queue.cc:237] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:21.664676 31528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "474ef75c3b7c4b13af5047a850a04d07" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35"
I20260808 01:48:21.671663 31528 raft_consensus.cc:3060] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.680426 31528 raft_consensus.cc:2468] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 1.
I20260808 01:48:21.683611 31527 raft_consensus.cc:2468] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 1.
I20260808 01:48:21.686211 31738 leader_election.cc:304] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, dd45cf41d4134d1e84318ca9aee76f35; no voters:
I20260808 01:48:21.688561 32058 raft_consensus.cc:2804] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:21.694053 31528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35" is_pre_election: true
I20260808 01:48:21.694733 31528 raft_consensus.cc:2468] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 0.
I20260808 01:48:21.706682 31804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "dd45cf41d4134d1e84318ca9aee76f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f7f87d8874aa2adab8d5e011cf479" is_pre_election: true
I20260808 01:48:21.707523 31804 raft_consensus.cc:2468] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd45cf41d4134d1e84318ca9aee76f35 in term 0.
I20260808 01:48:21.709857 32058 raft_consensus.cc:697] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Becoming Leader. State: Replica: 123f7f87d8874aa2adab8d5e011cf479, State: Running, Role: LEADER
I20260808 01:48:21.710695 32043 raft_consensus.cc:515] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.710739 32058 consensus_queue.cc:237] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:21.711974 31377 catalog_manager.cc:5697] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:21.715889 31946 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "dd45cf41d4134d1e84318ca9aee76f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805e196f2b1e48e2a598169ff776c113" is_pre_election: true
I20260808 01:48:21.720098 31664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6c1a2fdd3214f9c8cf5450f73b05e5b" candidate_uuid: "123f7f87d8874aa2adab8d5e011cf479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c"
I20260808 01:48:21.721058 31664 raft_consensus.cc:3060] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.721587 32062 leader_election.cc:290] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [CANDIDATE]: Term 1 election: Requested vote from peers dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955), b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495)
I20260808 01:48:21.716203 31665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "dd45cf41d4134d1e84318ca9aee76f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c" is_pre_election: true
I20260808 01:48:21.721143 32043 leader_election.cc:290] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 election: Requested vote from peers 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353), dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
I20260808 01:48:21.717465 31666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6b4daa349484e79875550e1f976021d" candidate_uuid: "dd45cf41d4134d1e84318ca9aee76f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f387bcc2a042bea411330031f4cb3c" is_pre_election: true
I20260808 01:48:21.727237 31528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45cf41d4134d1e84318ca9aee76f35"
I20260808 01:48:21.727483 31665 raft_consensus.cc:2393] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd45cf41d4134d1e84318ca9aee76f35 in current term 1: Already voted for candidate b3f387bcc2a042bea411330031f4cb3c in this term.
I20260808 01:48:21.727802 31528 raft_consensus.cc:3060] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.730176 31666 raft_consensus.cc:2393] T a6b4daa349484e79875550e1f976021d P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd45cf41d4134d1e84318ca9aee76f35 in current term 1: Already voted for candidate 805e196f2b1e48e2a598169ff776c113 in this term.
I20260808 01:48:21.732141 31463 leader_election.cc:304] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd45cf41d4134d1e84318ca9aee76f35; no voters: 805e196f2b1e48e2a598169ff776c113, b3f387bcc2a042bea411330031f4cb3c
I20260808 01:48:21.733310 32060 raft_consensus.cc:3060] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.734535 31528 raft_consensus.cc:2468] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 1.
I20260808 01:48:21.739521 31599 leader_election.cc:304] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3f387bcc2a042bea411330031f4cb3c, dd45cf41d4134d1e84318ca9aee76f35; no voters:
I20260808 01:48:21.740649 32043 raft_consensus.cc:2804] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:21.741196 32043 raft_consensus.cc:697] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEADER]: Becoming Leader. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Running, Role: LEADER
I20260808 01:48:21.742043 32043 consensus_queue.cc:237] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:21.756594 32060 raft_consensus.cc:2749] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:21.762923 31664 raft_consensus.cc:2468] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f7f87d8874aa2adab8d5e011cf479 in term 1.
I20260808 01:48:21.764360 31462 leader_election.cc:304] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123f7f87d8874aa2adab8d5e011cf479, dd45cf41d4134d1e84318ca9aee76f35; no voters: b3f387bcc2a042bea411330031f4cb3c
I20260808 01:48:21.765805 32060 raft_consensus.cc:2764] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:48:21.766206 31378 catalog_manager.cc:5697] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b3f387bcc2a042bea411330031f4cb3c (127.29.180.66). New cstate: current_term: 1 leader_uuid: "b3f387bcc2a042bea411330031f4cb3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:21.772666 31804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14ed40bb7c6c4b5fa28e64dfcacf50f8" candidate_uuid: "b3f387bcc2a042bea411330031f4cb3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123f7f87d8874aa2adab8d5e011cf479"
I20260808 01:48:21.773490 31804 raft_consensus.cc:3060] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:21.787889 31804 raft_consensus.cc:2468] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f387bcc2a042bea411330031f4cb3c in term 1.
I20260808 01:48:21.791713 31377 catalog_manager.cc:5697] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:21.881331 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31440
I20260808 01:48:21.911424 32043 consensus_queue.cc:1048] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:21.936422 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=file
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:42955
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:41127
--webserver_port=33469
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:21.946790 32043 consensus_queue.cc:1048] T f165e239f6fd412d8ba2c66079f53543 P b3f387bcc2a042bea411330031f4cb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:48:21.950461 31878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111)
I20260808 01:48:21.954087 32035 consensus_queue.cc:1048] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:21.972107 31878 consensus_peers.cc:597] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:22.137130 32058 consensus_queue.cc:1048] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:22.143344 31738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111)
W20260808 01:48:22.152138 31738 consensus_peers.cc:597] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:22.155002 31738 consensus_peers.cc:597] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:22.172086 31599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111)
W20260808 01:48:22.175940 31599 consensus_peers.cc:597] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:22.190387 32043 consensus_queue.cc:1048] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P b3f387bcc2a042bea411330031f4cb3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:22.235630 32058 consensus_queue.cc:1048] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:22.335343 32087 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:22.335929 32087 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:22.336552 32087 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:22.373347 32087 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:22.374481 32087 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:22.417703 32087 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=file
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:42955
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=33469
--tserver_master_addrs=127.29.180.126:41127
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:22.419498 32087 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:48:22.420722 32087 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:48:22.420928 32087 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:48:22.422178 32087 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:22.436226 32106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:23.837338 32105 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 32087
W20260808 01:48:24.143386 32087 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.706s user 0.602s sys 0.905s
W20260808 01:48:22.437659 32107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:24.144802 32087 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.708s user 0.602s sys 0.905s
W20260808 01:48:24.145892 32110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:24.148872 32108 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1706 milliseconds
I20260808 01:48:24.148924 32087 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:24.150292 32087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:24.152496 32087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:24.153879 32087 hybrid_clock.cc:648] HybridClock initialized: now 1786153704153826 us; error 46 us; skew 500 ppm
I20260808 01:48:24.154765 32087 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:24.161559 32087 webserver.cc:533] Webserver started at http://127.29.180.65:33469/ using document root <none> and password file <none>
I20260808 01:48:24.163343 32087 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:24.163632 32087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:48:24.166224 32087 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:48:24.171257 32087 env_posix.cc:431] I/O error, context: INJECTED FAILURE
W20260808 01:48:24.184870 32087 data_dirs.cc:323] instance /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data/block_manager_instance has unknown UUID bb1ffac17835492191bf3d9cbfe2d379
I20260808 01:48:24.186129 32087 fs_manager.cc:714] Time spent opening directory manager: real 0.017s user 0.008s sys 0.011s
I20260808 01:48:24.188325 32087 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:24.188552 32087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "dd45cf41d4134d1e84318ca9aee76f35"
format_stamp: "Formatted at 2026-08-08 01:48:12 on dist-test-slave-p3t5"
I20260808 01:48:24.190691 32087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Did not check for full LBM containers with extra space
Did not check for incomplete LBM containers
Did not check for malformed LBM records
Did not check for misaligned LBM blocks
Did not check for partial LBM records
Did not check for corrupted LBM rdb records
I20260808 01:48:24.246601 32087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:24.248282 32087 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:24.248757 32087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:24.251521 32087 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:24.258716 32135 ts_tablet_manager.cc:548] Loading tablet metadata (0/4 complete)
I20260808 01:48:24.279333 32087 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260808 01:48:24.279592 32087 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.023s user 0.003s sys 0.007s
I20260808 01:48:24.279899 32087 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
I20260808 01:48:24.286406 32138 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
E20260808 01:48:24.287230 32138 ts_tablet_manager.cc:1425] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260808 01:48:24.287590 32138 ts_tablet_manager.cc:1403] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 01:48:24.287905 32138 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:24.288246 32138 raft_consensus.cc:2243] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.291215 32138 raft_consensus.cc:2272] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.292268 32137 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
E20260808 01:48:24.292716 32137 ts_tablet_manager.cc:1425] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory
I20260808 01:48:24.292950 32137 ts_tablet_manager.cc:1403] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 01:48:24.293179 32137 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:24.293375 32137 raft_consensus.cc:2243] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.293679 32137 raft_consensus.cc:2272] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.296015 32137 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:24.299561 32087 ts_tablet_manager.cc:616] Registered 4 tablets
I20260808 01:48:24.299835 32087 ts_tablet_manager.cc:595] Time spent register tablets: real 0.020s user 0.018s sys 0.000s
I20260808 01:48:24.299888 32138 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap starting.
I20260808 01:48:24.361783 32137 log.cc:826] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:24.382128 32137 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:24.383795 32137 tablet_bootstrap.cc:492] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap complete.
I20260808 01:48:24.385931 32137 ts_tablet_manager.cc:1403] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.090s user 0.052s sys 0.029s
I20260808 01:48:24.392469 32138 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:24.394476 32138 tablet_bootstrap.cc:492] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Bootstrap complete.
I20260808 01:48:24.396622 32138 ts_tablet_manager.cc:1403] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Time spent bootstrapping tablet: real 0.097s user 0.050s sys 0.030s
I20260808 01:48:24.429730 32137 raft_consensus.cc:359] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:24.429809 32138 raft_consensus.cc:359] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:24.431335 32137 raft_consensus.cc:740] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:24.431437 32138 raft_consensus.cc:740] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45cf41d4134d1e84318ca9aee76f35, State: Initialized, Role: FOLLOWER
I20260808 01:48:24.432854 32137 consensus_queue.cc:260] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } }
I20260808 01:48:24.433013 32138 consensus_queue.cc:260] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "dd45cf41d4134d1e84318ca9aee76f35" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 42955 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } }
I20260808 01:48:24.440466 32138 ts_tablet_manager.cc:1434] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.043s user 0.007s sys 0.008s
I20260808 01:48:24.441399 32137 ts_tablet_manager.cc:1434] T a6b4daa349484e79875550e1f976021d P dd45cf41d4134d1e84318ca9aee76f35: Time spent starting tablet: real 0.055s user 0.038s sys 0.013s
W20260808 01:48:24.450341 31878 consensus_peers.cc:597] T a6b4daa349484e79875550e1f976021d P 805e196f2b1e48e2a598169ff776c113 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:24.525350 32087 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:42955
I20260808 01:48:24.525619 32252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:42955 every 8 connection(s)
I20260808 01:48:24.528394 32087 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:48:24.532196 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 32087
W20260808 01:48:24.532719 32254 tablet.cc:2406] T 14ed40bb7c6c4b5fa28e64dfcacf50f8 P dd45cf41d4134d1e84318ca9aee76f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:24.551812 32253 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41127
I20260808 01:48:24.552297 32253 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:24.553408 32253 heartbeater.cc:507] Master 127.29.180.126:41127 requested a full tablet report, sending...
I20260808 01:48:24.558101 31378 ts_manager.cc:194] Re-registered known tserver with Master: dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955)
W20260808 01:48:24.561834 31378 catalog_manager.cc:5610] Tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (table test-workload [id=24b28d797daa4674942b43d523058529]) has failed on TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): IO error: some tablet data is in a failed directory
W20260808 01:48:24.562621 31378 catalog_manager.cc:5610] Tablet 474ef75c3b7c4b13af5047a850a04d07 (table test-workload [id=24b28d797daa4674942b43d523058529]) has failed on TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): IO error: some tablet data is in a failed directory
I20260808 01:48:24.568583 31378 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:54393
I20260808 01:48:24.673123 31738 consensus_queue.cc:579] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: The tablet replica hosted on peer dd45cf41d4134d1e84318ca9aee76f35 has failed
W20260808 01:48:24.674134 31738 consensus_peers.cc:597] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:24.681174 32253 heartbeater.cc:499] Master 127.29.180.126:41127 was elected leader, sending a full tablet report...
I20260808 01:48:24.695417 31804 consensus_queue.cc:237] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } }
I20260808 01:48:24.712126 31666 raft_consensus.cc:1275] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:24.715688 32058 consensus_queue.cc:1048] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:48:24.732296 31378 catalog_manager.cc:5610] Tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (table test-workload [id=24b28d797daa4674942b43d523058529]) has failed on TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): IO error: some tablet data is in a failed directory
W20260808 01:48:24.733038 31378 catalog_manager.cc:5610] Tablet 474ef75c3b7c4b13af5047a850a04d07 (table test-workload [id=24b28d797daa4674942b43d523058529]) has failed on TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): IO error: some tablet data is in a failed directory
I20260808 01:48:24.734848 32156 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } }
I20260808 01:48:24.741232 31664 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } }
I20260808 01:48:24.753008 31365 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:48:24.759684 31378 catalog_manager.cc:5697] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 reported cstate change: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:24.799090 31805 consensus_queue.cc:237] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } }
W20260808 01:48:24.816941 31739 consensus_peers.cc:597] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 -> Peer 805e196f2b1e48e2a598169ff776c113 (127.29.180.68:45233): Couldn't send request to peer 805e196f2b1e48e2a598169ff776c113. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c1a2fdd3214f9c8cf5450f73b05e5b. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:24.819923 32185 tablet_service.cc:1558] Processing DeleteTablet for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b with delete_type TABLET_DATA_TOMBSTONED (TS dd45cf41d4134d1e84318ca9aee76f35 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:48854
I20260808 01:48:24.820685 31664 raft_consensus.cc:1275] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:48:24.823258 32156 consensus_queue.cc:1048] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:24.825034 32277 ts_tablet_manager.cc:1916] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:24.833536 32058 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } } }
I20260808 01:48:24.836184 31664 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } } }
I20260808 01:48:24.857447 31365 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:48:24.872824 32277 ts_tablet_manager.cc:1929] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:24.873463 32277 log.cc:1199] T e6c1a2fdd3214f9c8cf5450f73b05e5b P dd45cf41d4134d1e84318ca9aee76f35: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/e6c1a2fdd3214f9c8cf5450f73b05e5b
I20260808 01:48:24.880790 31363 catalog_manager.cc:5028] TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): tablet e6c1a2fdd3214f9c8cf5450f73b05e5b (table test-workload [id=24b28d797daa4674942b43d523058529]) successfully deleted
I20260808 01:48:24.880340 31378 catalog_manager.cc:5697] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c reported cstate change: config changed from index 2 to 3, NON_VOTER 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } } }
I20260808 01:48:24.935128 31738 consensus_queue.cc:579] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: The tablet replica hosted on peer dd45cf41d4134d1e84318ca9aee76f35 has failed
W20260808 01:48:24.935659 31738 consensus_peers.cc:597] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 -> Peer dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): Couldn't send request to peer dd45cf41d4134d1e84318ca9aee76f35. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:24.947855 31805 consensus_queue.cc:237] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } }
I20260808 01:48:24.954027 31946 raft_consensus.cc:1275] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:24.955499 32156 consensus_queue.cc:1048] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:24.963260 32278 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } }
I20260808 01:48:24.965384 31946 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } }
I20260808 01:48:24.976688 31365 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 474ef75c3b7c4b13af5047a850a04d07 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:48:24.980917 31377 catalog_manager.cc:5697] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 reported cstate change: config changed from index -1 to 2, VOTER dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } }
I20260808 01:48:24.994490 32185 tablet_service.cc:1558] Processing DeleteTablet for tablet 474ef75c3b7c4b13af5047a850a04d07 with delete_type TABLET_DATA_TOMBSTONED (TS dd45cf41d4134d1e84318ca9aee76f35 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:48854
I20260808 01:48:24.995716 32277 ts_tablet_manager.cc:1916] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:24.999297 31805 consensus_queue.cc:237] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } }
I20260808 01:48:25.007345 31946 raft_consensus.cc:1275] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:48:25.009521 32278 consensus_queue.cc:1048] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:48:25.011763 31741 consensus_peers.cc:597] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 -> Peer b3f387bcc2a042bea411330031f4cb3c (127.29.180.66:42495): Couldn't send request to peer b3f387bcc2a042bea411330031f4cb3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 474ef75c3b7c4b13af5047a850a04d07. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:25.013675 32277 ts_tablet_manager.cc:1929] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:25.014065 32277 log.cc:1199] T 474ef75c3b7c4b13af5047a850a04d07 P dd45cf41d4134d1e84318ca9aee76f35: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/474ef75c3b7c4b13af5047a850a04d07
I20260808 01:48:25.015681 31363 catalog_manager.cc:5028] TS dd45cf41d4134d1e84318ca9aee76f35 (127.29.180.65:42955): tablet 474ef75c3b7c4b13af5047a850a04d07 (table test-workload [id=24b28d797daa4674942b43d523058529]) successfully deleted
I20260808 01:48:25.018343 32058 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } } }
I20260808 01:48:25.020735 31946 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } } }
I20260808 01:48:25.037206 31365 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 474ef75c3b7c4b13af5047a850a04d07 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:48:25.040720 31377 catalog_manager.cc:5697] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 reported cstate change: config changed from index 2 to 3, NON_VOTER b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) added. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } } }
I20260808 01:48:25.171939 31784 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260808 01:48:25.190673 31924 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:48:25.196584 32185 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:25.201076 31643 tablet_service.cc:1467] Tablet server has 2 leaders and 0 scanners
I20260808 01:48:25.389070 32306 ts_tablet_manager.cc:933] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Initiating tablet copy from peer 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353)
I20260808 01:48:25.424474 32306 tablet_copy_client.cc:323] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:33353
I20260808 01:48:25.460887 31825 tablet_copy_service.cc:140] P 123f7f87d8874aa2adab8d5e011cf479: Received BeginTabletCopySession request for tablet e6c1a2fdd3214f9c8cf5450f73b05e5b from peer 805e196f2b1e48e2a598169ff776c113 ({username='slave'} at 127.29.180.68:51999)
I20260808 01:48:25.461750 31825 tablet_copy_service.cc:161] P 123f7f87d8874aa2adab8d5e011cf479: Beginning new tablet copy session on tablet e6c1a2fdd3214f9c8cf5450f73b05e5b from peer 805e196f2b1e48e2a598169ff776c113 at {username='slave'} at 127.29.180.68:51999: session id = 805e196f2b1e48e2a598169ff776c113-e6c1a2fdd3214f9c8cf5450f73b05e5b
I20260808 01:48:25.474609 31825 tablet_copy_source_session.cc:215] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:25.515020 32306 tablet_copy_client.cc:806] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:25.515663 32306 tablet_copy_client.cc:670] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:25.520954 32306 tablet_copy_client.cc:538] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:25.534370 32306 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Bootstrap starting.
I20260808 01:48:25.583725 32310 ts_tablet_manager.cc:933] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Initiating tablet copy from peer 123f7f87d8874aa2adab8d5e011cf479 (127.29.180.67:33353)
I20260808 01:48:25.598361 32310 tablet_copy_client.cc:323] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:33353
I20260808 01:48:25.601125 31825 tablet_copy_service.cc:140] P 123f7f87d8874aa2adab8d5e011cf479: Received BeginTabletCopySession request for tablet 474ef75c3b7c4b13af5047a850a04d07 from peer b3f387bcc2a042bea411330031f4cb3c ({username='slave'} at 127.29.180.66:37635)
I20260808 01:48:25.602034 31825 tablet_copy_service.cc:161] P 123f7f87d8874aa2adab8d5e011cf479: Beginning new tablet copy session on tablet 474ef75c3b7c4b13af5047a850a04d07 from peer b3f387bcc2a042bea411330031f4cb3c at {username='slave'} at 127.29.180.66:37635: session id = b3f387bcc2a042bea411330031f4cb3c-474ef75c3b7c4b13af5047a850a04d07
I20260808 01:48:25.611037 31825 tablet_copy_source_session.cc:215] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:25.661128 32306 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:25.674858 32306 tablet_bootstrap.cc:492] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Bootstrap complete.
I20260808 01:48:25.675664 32306 ts_tablet_manager.cc:1403] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Time spent bootstrapping tablet: real 0.142s user 0.023s sys 0.022s
I20260808 01:48:25.679205 32306 raft_consensus.cc:359] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } }
I20260808 01:48:25.680012 32306 raft_consensus.cc:740] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 805e196f2b1e48e2a598169ff776c113, State: Initialized, Role: LEARNER
I20260808 01:48:25.683085 32310 tablet_copy_client.cc:806] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:25.684125 32310 tablet_copy_client.cc:670] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: tablet copy: Starting download of 1 WAL segments...
E20260808 01:48:25.688165 32248 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:25.695712 32310 tablet_copy_client.cc:538] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:25.714087 32310 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Bootstrap starting.
I20260808 01:48:25.680745 32306 consensus_queue.cc:260] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: true } }
I20260808 01:48:25.729722 32306 ts_tablet_manager.cc:1434] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113: Time spent starting tablet: real 0.054s user 0.006s sys 0.003s
I20260808 01:48:25.731961 31825 tablet_copy_service.cc:342] P 123f7f87d8874aa2adab8d5e011cf479: Request end of tablet copy session 805e196f2b1e48e2a598169ff776c113-e6c1a2fdd3214f9c8cf5450f73b05e5b received from {username='slave'} at 127.29.180.68:51999
I20260808 01:48:25.732471 31825 tablet_copy_service.cc:434] P 123f7f87d8874aa2adab8d5e011cf479: ending tablet copy session 805e196f2b1e48e2a598169ff776c113-e6c1a2fdd3214f9c8cf5450f73b05e5b on tablet e6c1a2fdd3214f9c8cf5450f73b05e5b with peer 805e196f2b1e48e2a598169ff776c113
I20260808 01:48:25.768232 32310 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:25.769260 32310 tablet_bootstrap.cc:492] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Bootstrap complete.
I20260808 01:48:25.770110 32310 ts_tablet_manager.cc:1403] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.016s
I20260808 01:48:25.775840 32310 raft_consensus.cc:359] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } }
I20260808 01:48:25.777173 32310 raft_consensus.cc:740] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3f387bcc2a042bea411330031f4cb3c, State: Initialized, Role: LEARNER
I20260808 01:48:25.778004 32310 consensus_queue.cc:260] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: true } }
I20260808 01:48:25.781536 32310 ts_tablet_manager.cc:1434] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260808 01:48:25.791814 31825 tablet_copy_service.cc:342] P 123f7f87d8874aa2adab8d5e011cf479: Request end of tablet copy session b3f387bcc2a042bea411330031f4cb3c-474ef75c3b7c4b13af5047a850a04d07 received from {username='slave'} at 127.29.180.66:37635
I20260808 01:48:25.792404 31825 tablet_copy_service.cc:434] P 123f7f87d8874aa2adab8d5e011cf479: ending tablet copy session b3f387bcc2a042bea411330031f4cb3c-474ef75c3b7c4b13af5047a850a04d07 on tablet 474ef75c3b7c4b13af5047a850a04d07 with peer b3f387bcc2a042bea411330031f4cb3c
I20260808 01:48:25.875620 31946 raft_consensus.cc:1217] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
df3e745e81ab4066b37ecdf9252eaf54 | 127.29.180.126:41127 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:40717 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
123f7f87d8874aa2adab8d5e011cf479 | 127.29.180.67:33353 | HEALTHY | <none> | 2 | 0
805e196f2b1e48e2a598169ff776c113 | 127.29.180.68:45233 | HEALTHY | <none> | 1 | 0
b3f387bcc2a042bea411330031f4cb3c | 127.29.180.66:42495 | HEALTHY | <none> | 2 | 0
dd45cf41d4134d1e84318ca9aee76f35 | 127.29.180.65:42955 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:42955
env_inject_eio | 1 | hidden | 127.29.180.65:42955
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:42955
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:42955
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:42495
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:33353
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:45233
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:42955
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:42495
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:33353
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:45233
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:40717 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | dd45cf41d4134d1e84318ca9aee76f35 | 127.29.180.65:42955 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:48:25.902540 30417 log_verifier.cc:126] Checking tablet 14ed40bb7c6c4b5fa28e64dfcacf50f8
I20260808 01:48:26.000427 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 14ed40bb7c6c4b5fa28e64dfcacf50f8
I20260808 01:48:26.000743 30417 log_verifier.cc:126] Checking tablet 474ef75c3b7c4b13af5047a850a04d07
I20260808 01:48:26.093111 31664 raft_consensus.cc:1217] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260808 01:48:26.105309 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 474ef75c3b7c4b13af5047a850a04d07
I20260808 01:48:26.105588 30417 log_verifier.cc:126] Checking tablet a6b4daa349484e79875550e1f976021d
I20260808 01:48:26.198055 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a6b4daa349484e79875550e1f976021d
I20260808 01:48:26.198336 30417 log_verifier.cc:126] Checking tablet e6c1a2fdd3214f9c8cf5450f73b05e5b
I20260808 01:48:26.267650 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e6c1a2fdd3214f9c8cf5450f73b05e5b
I20260808 01:48:26.267946 30417 log_verifier.cc:126] Checking tablet f165e239f6fd412d8ba2c66079f53543
I20260808 01:48:26.363855 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f165e239f6fd412d8ba2c66079f53543
I20260808 01:48:26.499193 32316 raft_consensus.cc:1064] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479: attempting to promote NON_VOTER 805e196f2b1e48e2a598169ff776c113 to VOTER
I20260808 01:48:26.502575 32316 consensus_queue.cc:237] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false } }
I20260808 01:48:26.510815 31664 raft_consensus.cc:1275] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:26.510862 31946 raft_consensus.cc:1275] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [term 1 LEARNER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:26.513132 32058 consensus_queue.cc:1048] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:26.514457 32317 consensus_queue.cc:1048] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:26.528424 32058 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false } } }
I20260808 01:48:26.530508 31664 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false } } }
I20260808 01:48:26.533509 31946 raft_consensus.cc:2955] T e6c1a2fdd3214f9c8cf5450f73b05e5b P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false } } }
I20260808 01:48:26.542320 31378 catalog_manager.cc:5697] T e6c1a2fdd3214f9c8cf5450f73b05e5b P b3f387bcc2a042bea411330031f4cb3c reported cstate change: config changed from index 3 to 4, 805e196f2b1e48e2a598169ff776c113 (127.29.180.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } } peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } attrs { promote: false } } }
I20260808 01:48:26.715204 32058 raft_consensus.cc:1064] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479: attempting to promote NON_VOTER b3f387bcc2a042bea411330031f4cb3c to VOTER
I20260808 01:48:26.716897 32058 consensus_queue.cc:237] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false } }
I20260808 01:48:26.721902 31664 raft_consensus.cc:1275] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [term 1 LEARNER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:26.722066 31946 raft_consensus.cc:1275] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Refusing update from remote peer 123f7f87d8874aa2adab8d5e011cf479: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:26.723292 32058 consensus_queue.cc:1048] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:26.724022 32316 consensus_queue.cc:1048] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:26.735535 31946 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false } } }
I20260808 01:48:26.732668 32317 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P 123f7f87d8874aa2adab8d5e011cf479 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false } } }
I20260808 01:48:26.746906 31664 raft_consensus.cc:2955] T 474ef75c3b7c4b13af5047a850a04d07 P b3f387bcc2a042bea411330031f4cb3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false } } }
I20260808 01:48:26.759641 31379 catalog_manager.cc:5697] T 474ef75c3b7c4b13af5047a850a04d07 P 805e196f2b1e48e2a598169ff776c113 reported cstate change: config changed from index 3 to 4, b3f387bcc2a042bea411330031f4cb3c (127.29.180.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "123f7f87d8874aa2adab8d5e011cf479" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "123f7f87d8874aa2adab8d5e011cf479" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33353 } } peers { permanent_uuid: "805e196f2b1e48e2a598169ff776c113" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 45233 } } peers { permanent_uuid: "b3f387bcc2a042bea411330031f4cb3c" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42495 } attrs { promote: false } } }
I20260808 01:48:26.955873 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 32087
I20260808 01:48:27.030949 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31579
W20260808 01:48:27.083039 31741 connection.cc:573] client connection to 127.29.180.66:42495 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 01:48:27.094087 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31717
W20260808 01:48:27.137447 31878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:42955: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20260808 01:48:27.138355 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31854
I20260808 01:48:27.175534 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 31341
2026-08-08T01:48:27Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1 (20017 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2
2026-08-08T01:48:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:48:27Z Disabled control of system clock
I20260808 01:48:27.309355 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41303
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:41303
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:48:27.654858 32351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:27.655547 32351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:27.655964 32351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:27.690006 32351 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:27.690346 32351 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:48:27.690558 32351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:27.690773 32351 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:48:27.690976 32351 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:48:27.729429 32351 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:41303
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:27.730840 32351 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:27.732550 32351 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:27.744199 32357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:27.745298 32358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:29.022535 32360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:29.024995 32359 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1275 milliseconds
I20260808 01:48:29.025143 32351 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:29.026619 32351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:29.030082 32351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:29.031539 32351 hybrid_clock.cc:648] HybridClock initialized: now 1786153709031483 us; error 68 us; skew 500 ppm
I20260808 01:48:29.032415 32351 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:29.039645 32351 webserver.cc:533] Webserver started at http://127.29.180.126:45475/ using document root <none> and password file <none>
I20260808 01:48:29.041162 32351 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:29.041368 32351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:29.042059 32351 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:29.046644 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.047814 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.048686 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.049527 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.050460 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.051324 32351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.075896 32351 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.019s sys 0.006s
I20260808 01:48:29.085075 32372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:29.085829 32372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:29.086520 32372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:29.086756 32373 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:29.086879 32372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:29.089484 32351 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.008s
I20260808 01:48:29.089901 32351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "89c7916fdcd048d58b330891aa1c95db"
format_stamp: "Formatted at 2026-08-08 01:48:29 on dist-test-slave-p3t5"
I20260808 01:48:29.090287 32351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:29.158311 32351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:29.159858 32351 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:29.160367 32351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:29.252578 32351 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:41303
I20260808 01:48:29.252657 32426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:41303 every 8 connection(s)
I20260808 01:48:29.255703 32351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:48:29.258129 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 32351
I20260808 01:48:29.258991 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:48:29.291105 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db: Bootstrap starting.
I20260808 01:48:29.297588 32427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:29.299665 32427 log.cc:826] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:29.305392 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db: No bootstrap required, opened a new log
I20260808 01:48:29.326419 32427 raft_consensus.cc:359] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } }
I20260808 01:48:29.327162 32427 raft_consensus.cc:385] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:29.327484 32427 raft_consensus.cc:740] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c7916fdcd048d58b330891aa1c95db, State: Initialized, Role: FOLLOWER
I20260808 01:48:29.328783 32427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } }
I20260808 01:48:29.329530 32427 raft_consensus.cc:399] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:29.329970 32427 raft_consensus.cc:493] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:29.330339 32427 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:29.335175 32427 raft_consensus.cc:515] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } }
I20260808 01:48:29.335947 32427 leader_election.cc:304] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c7916fdcd048d58b330891aa1c95db; no voters:
I20260808 01:48:29.338397 32427 leader_election.cc:290] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:48:29.338897 32432 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:29.343119 32432 raft_consensus.cc:697] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [term 1 LEADER]: Becoming Leader. State: Replica: 89c7916fdcd048d58b330891aa1c95db, State: Running, Role: LEADER
I20260808 01:48:29.344225 32427 sys_catalog.cc:565] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:29.344141 32432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } }
I20260808 01:48:29.358861 32433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c7916fdcd048d58b330891aa1c95db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } } }
I20260808 01:48:29.360180 32433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:29.358605 32434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c7916fdcd048d58b330891aa1c95db. Latest consensus state: current_term: 1 leader_uuid: "89c7916fdcd048d58b330891aa1c95db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c7916fdcd048d58b330891aa1c95db" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41303 } } }
I20260808 01:48:29.362246 32434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:29.372175 32440 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:29.386811 32440 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:29.406133 32440 catalog_manager.cc:1383] Generated new cluster ID: 44ff355179b148659b4dd3bf3d3992ff
I20260808 01:48:29.406450 32440 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:29.423631 32440 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:29.425694 32440 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:29.440335 32440 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 89c7916fdcd048d58b330891aa1c95db: Generated new TSK 0
I20260808 01:48:29.441642 32440 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:29.464461 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--builtin_ntp_servers=127.29.180.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:48:29.814743 32453 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:29.815333 32453 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:29.815881 32453 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:29.850991 32453 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:29.851416 32453 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:29.852236 32453 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:29.890802 32453 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:29.892320 32453 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:29.894284 32453 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:29.907940 32459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:31.311239 32458 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 32453
W20260808 01:48:31.454510 32458 kernel_stack_watchdog.cc:198] Thread 32453 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:31.459088 32453 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.550s user 0.442s sys 0.936s
W20260808 01:48:31.459853 32453 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.551s user 0.443s sys 0.937s
W20260808 01:48:31.460812 32461 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1551 milliseconds
W20260808 01:48:29.909235 32460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:31.461884 32462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:31.462548 32453 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:31.468192 32453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:31.471936 32453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:31.473583 32453 hybrid_clock.cc:648] HybridClock initialized: now 1786153711473446 us; error 119 us; skew 500 ppm
I20260808 01:48:31.474841 32453 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:31.484551 32453 webserver.cc:533] Webserver started at http://127.29.180.65:39497/ using document root <none> and password file <none>
I20260808 01:48:31.486495 32453 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:31.486795 32453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:31.487483 32453 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:31.492262 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.493901 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.495415 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.496857 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.498270 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.499630 32453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
W20260808 01:48:31.531419 32423 debug-util.cc:398] Leaking SignalData structure 0x7b0800074fe0 after lost signal to thread 32352
W20260808 01:48:31.532231 32423 debug-util.cc:398] Leaking SignalData structure 0x7b08000419e0 after lost signal to thread 32426
I20260808 01:48:31.543099 32453 fs_manager.cc:696] Time spent creating directory manager: real 0.043s user 0.020s sys 0.017s
I20260808 01:48:31.556624 32473 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.557430 32473 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.559077 32474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.559646 32474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.560074 32474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.562929 32453 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.011s sys 0.000s
I20260808 01:48:31.563370 32453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:31.563933 32453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:31.631393 32453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:31.633692 32453 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:31.634367 32453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:31.638863 32453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:31.645641 32453 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:31.645947 32453 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 01:48:31.646380 32453 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:31.646652 32453 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:31.832727 32453 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:34975
I20260808 01:48:31.832859 32590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:34975 every 8 connection(s)
I20260808 01:48:31.837083 32453 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:48:31.840507 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 32453
I20260808 01:48:31.841009 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:48:31.850270 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--builtin_ntp_servers=127.29.180.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:48:31.897207 32591 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41303
I20260808 01:48:31.897859 32591 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:31.899319 32591 heartbeater.cc:507] Master 127.29.180.126:41303 requested a full tablet report, sending...
I20260808 01:48:31.902989 32391 ts_manager.cc:194] Registered new tserver with Master: 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975)
I20260808 01:48:31.905884 32391 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:46267
W20260808 01:48:32.377565 32595 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:32.378222 32595 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:32.378755 32595 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:32.415115 32595 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:32.415525 32595 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:32.416340 32595 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:48:32.456717 32595 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:32.458292 32595 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:32.460204 32595 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:32.474387 32601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:32.910836 32591 heartbeater.cc:499] Master 127.29.180.126:41303 was elected leader, sending a full tablet report...
W20260808 01:48:33.878142 32600 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 32595
W20260808 01:48:34.254299 32595 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.779s user 0.596s sys 1.148s
W20260808 01:48:34.255394 32595 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.781s user 0.596s sys 1.148s
W20260808 01:48:34.255637 32603 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1780 milliseconds
W20260808 01:48:32.476727 32602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:34.256675 32595 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260808 01:48:34.256647 32604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:34.261585 32595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:34.264453 32595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:34.265937 32595 hybrid_clock.cc:648] HybridClock initialized: now 1786153714265875 us; error 52 us; skew 500 ppm
I20260808 01:48:34.267133 32595 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:34.275691 32595 webserver.cc:533] Webserver started at http://127.29.180.66:40245/ using document root <none> and password file <none>
I20260808 01:48:34.278115 32595 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:34.278461 32595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:34.279435 32595 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:34.286938 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.288715 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.290091 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.291419 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.292750 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.294143 32595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.327340 32595 fs_manager.cc:696] Time spent creating directory manager: real 0.032s user 0.022s sys 0.010s
I20260808 01:48:34.336330 32615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.337014 32615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.337939 32615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.338092 32616 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.338421 32615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.341039 32595 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.008s sys 0.002s
I20260808 01:48:34.341418 32595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
format_stamp: "Formatted at 2026-08-08 01:48:34 on dist-test-slave-p3t5"
I20260808 01:48:34.341948 32595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:34.407928 32595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:34.409557 32595 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:34.410097 32595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:34.413009 32595 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:34.418012 32595 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:34.418263 32595 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.418548 32595 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:34.418730 32595 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:34.611816 32595 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:40163
I20260808 01:48:34.611943 32731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:40163 every 8 connection(s)
I20260808 01:48:34.615640 32595 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:48:34.619562 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 32595
I20260808 01:48:34.620076 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:48:34.640429 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--builtin_ntp_servers=127.29.180.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:48:34.648254 32732 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41303
I20260808 01:48:34.648856 32732 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:34.650520 32732 heartbeater.cc:507] Master 127.29.180.126:41303 requested a full tablet report, sending...
I20260808 01:48:34.653505 32391 ts_manager.cc:194] Registered new tserver with Master: aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:34.655258 32391 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:59617
W20260808 01:48:35.009009 32736 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:35.009656 32736 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:35.010198 32736 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:35.045053 32736 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:35.045512 32736 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:35.046537 32736 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:48:35.085408 32736 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:35.086933 32736 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:35.088687 32736 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:35.101233 32742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:35.659284 32732 heartbeater.cc:499] Master 127.29.180.126:41303 was elected leader, sending a full tablet report...
W20260808 01:48:35.103228 32743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:36.317436 32745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:36.319288 32744 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1212 milliseconds
I20260808 01:48:36.319396 32736 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:36.320699 32736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:36.323736 32736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:36.325280 32736 hybrid_clock.cc:648] HybridClock initialized: now 1786153716325244 us; error 49 us; skew 500 ppm
I20260808 01:48:36.326196 32736 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:36.333534 32736 webserver.cc:533] Webserver started at http://127.29.180.67:32923/ using document root <none> and password file <none>
I20260808 01:48:36.335124 32736 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:36.335392 32736 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:36.336081 32736 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:36.340636 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.341907 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.342810 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.343709 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.344589 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.345463 32736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.370814 32736 fs_manager.cc:696] Time spent creating directory manager: real 0.025s user 0.022s sys 0.001s
I20260808 01:48:36.379431 32756 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.380096 32756 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.381165 32756 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.381398 32757 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.381510 32756 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.383972 32736 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.002s
I20260808 01:48:36.384516 32736 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "c6bf8ae821ac41859a4aebb8ede00632"
format_stamp: "Formatted at 2026-08-08 01:48:36 on dist-test-slave-p3t5"
I20260808 01:48:36.384923 32736 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:36.437120 32736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:36.438688 32736 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:36.439137 32736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:36.441977 32736 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:36.446833 32736 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:36.447047 32736 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.447368 32736 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:36.447522 32736 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:36.597087 32736 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:33183
I20260808 01:48:36.597213 403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:33183 every 8 connection(s)
I20260808 01:48:36.599895 32736 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:48:36.610335 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 32736
I20260808 01:48:36.610934 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:48:36.632390 404 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41303
I20260808 01:48:36.632989 404 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:36.634378 404 heartbeater.cc:507] Master 127.29.180.126:41303 requested a full tablet report, sending...
I20260808 01:48:36.636605 32391 ts_manager.cc:194] Registered new tserver with Master: c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:36.637950 32391 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:53285
I20260808 01:48:36.649128 30417 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260808 01:48:36.725731 32391 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54092:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:48:36.802409 32663 tablet_service.cc:1511] Processing CreateTablet for tablet e319d6c4dac04f1b9856214abe251b74 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:36.806243 32665 tablet_service.cc:1511] Processing CreateTablet for tablet c5181f3f52b649e2847c9b9107d3f7f5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:36.806144 32520 tablet_service.cc:1511] Processing CreateTablet for tablet 6482d8c7bc374c88826a3e3f295c2021 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:36.808405 32521 tablet_service.cc:1511] Processing CreateTablet for tablet e319d6c4dac04f1b9856214abe251b74 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:36.811345 32522 tablet_service.cc:1511] Processing CreateTablet for tablet ef18b6b84f7b40419d7f568744c17b24 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:36.811388 337 tablet_service.cc:1511] Processing CreateTablet for tablet ef18b6b84f7b40419d7f568744c17b24 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:36.814322 336 tablet_service.cc:1511] Processing CreateTablet for tablet e319d6c4dac04f1b9856214abe251b74 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:36.814917 32523 tablet_service.cc:1511] Processing CreateTablet for tablet c5181f3f52b649e2847c9b9107d3f7f5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:36.816290 335 tablet_service.cc:1511] Processing CreateTablet for tablet 6482d8c7bc374c88826a3e3f295c2021 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:36.802377 32666 tablet_service.cc:1511] Processing CreateTablet for tablet b1fcf8b4e5b1403b97bad94682add9e5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:36.818001 338 tablet_service.cc:1511] Processing CreateTablet for tablet c5181f3f52b649e2847c9b9107d3f7f5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:36.818533 32662 tablet_service.cc:1511] Processing CreateTablet for tablet 6482d8c7bc374c88826a3e3f295c2021 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:36.819238 339 tablet_service.cc:1511] Processing CreateTablet for tablet b1fcf8b4e5b1403b97bad94682add9e5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:36.820863 32664 tablet_service.cc:1511] Processing CreateTablet for tablet ef18b6b84f7b40419d7f568744c17b24 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:36.827647 32524 tablet_service.cc:1511] Processing CreateTablet for tablet b1fcf8b4e5b1403b97bad94682add9e5 (DEFAULT_TABLE table=test-workload [id=cccf5de361c347b692a5096b0b0b56b4]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:36.903167 431 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Bootstrap starting.
I20260808 01:48:36.903270 434 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:36.914549 438 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Bootstrap starting.
I20260808 01:48:36.905345 430 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:36.906486 435 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:36.912742 433 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:36.917909 436 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:36.936299 440 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632: Bootstrap starting.
I20260808 01:48:36.920050 439 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Bootstrap starting.
I20260808 01:48:36.939533 443 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Bootstrap starting.
I20260808 01:48:36.939800 441 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632: Bootstrap starting.
I20260808 01:48:36.940384 430 tablet_bootstrap.cc:654] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.941875 442 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Bootstrap starting.
I20260808 01:48:36.922483 431 tablet_bootstrap.cc:654] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.934446 437 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Bootstrap starting.
I20260808 01:48:36.947454 433 tablet_bootstrap.cc:654] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.948376 431 log.cc:826] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:36.948379 430 log.cc:826] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:36.960961 439 tablet_bootstrap.cc:654] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.949741 436 tablet_bootstrap.cc:654] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.945581 444 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Bootstrap starting.
I20260808 01:48:36.976953 437 tablet_bootstrap.cc:654] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.979372 438 tablet_bootstrap.cc:654] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.980494 440 tablet_bootstrap.cc:654] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.980506 434 tablet_bootstrap.cc:654] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.984966 443 tablet_bootstrap.cc:654] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.985023 431 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: No bootstrap required, opened a new log
I20260808 01:48:36.985594 431 ts_tablet_manager.cc:1403] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent bootstrapping tablet: real 0.083s user 0.018s sys 0.006s
I20260808 01:48:36.986989 432 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632: Bootstrap starting.
I20260808 01:48:36.988765 444 tablet_bootstrap.cc:654] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.990912 436 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: No bootstrap required, opened a new log
I20260808 01:48:36.991515 436 ts_tablet_manager.cc:1403] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.006s
I20260808 01:48:36.960527 435 tablet_bootstrap.cc:654] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:36.964778 433 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: No bootstrap required, opened a new log
I20260808 01:48:37.001667 442 tablet_bootstrap.cc:654] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:37.002767 433 ts_tablet_manager.cc:1403] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.010s
I20260808 01:48:37.005559 443 log.cc:826] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:37.011698 441 tablet_bootstrap.cc:654] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:37.016419 437 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: No bootstrap required, opened a new log
I20260808 01:48:37.017041 437 ts_tablet_manager.cc:1403] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.009s
I20260808 01:48:37.018545 430 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: No bootstrap required, opened a new log
I20260808 01:48:37.019148 430 ts_tablet_manager.cc:1403] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.115s user 0.013s sys 0.009s
I20260808 01:48:37.029652 438 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: No bootstrap required, opened a new log
I20260808 01:48:37.030145 438 ts_tablet_manager.cc:1403] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent bootstrapping tablet: real 0.116s user 0.021s sys 0.005s
I20260808 01:48:37.032469 434 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: No bootstrap required, opened a new log
I20260808 01:48:37.032987 434 ts_tablet_manager.cc:1403] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.131s user 0.016s sys 0.000s
I20260808 01:48:37.033191 439 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: No bootstrap required, opened a new log
I20260808 01:48:37.033635 439 ts_tablet_manager.cc:1403] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent bootstrapping tablet: real 0.114s user 0.025s sys 0.043s
I20260808 01:48:37.039950 432 tablet_bootstrap.cc:654] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:37.039685 437 raft_consensus.cc:359] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.040707 437 raft_consensus.cc:385] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.041038 437 raft_consensus.cc:740] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.042161 431 raft_consensus.cc:359] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.042423 438 raft_consensus.cc:359] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.042917 431 raft_consensus.cc:385] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.043084 438 raft_consensus.cc:385] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.043223 431 raft_consensus.cc:740] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.043373 438 raft_consensus.cc:740] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.044344 431 consensus_queue.cc:260] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.044871 440 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632: No bootstrap required, opened a new log
I20260808 01:48:37.045401 440 ts_tablet_manager.cc:1403] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.003s
I20260808 01:48:37.047307 443 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: No bootstrap required, opened a new log
I20260808 01:48:37.045657 439 raft_consensus.cc:359] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.047652 439 raft_consensus.cc:385] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.047511 437 consensus_queue.cc:260] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.047941 439 raft_consensus.cc:740] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.050030 444 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: No bootstrap required, opened a new log
I20260808 01:48:37.050462 444 ts_tablet_manager.cc:1403] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.003s
I20260808 01:48:37.050144 439 consensus_queue.cc:260] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.051143 443 ts_tablet_manager.cc:1403] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.008s
I20260808 01:48:37.053925 444 raft_consensus.cc:359] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.054852 444 raft_consensus.cc:385] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.049543 435 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: No bootstrap required, opened a new log
I20260808 01:48:37.055203 444 raft_consensus.cc:740] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.044241 438 consensus_queue.cc:260] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.056403 435 ts_tablet_manager.cc:1403] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.004s
I20260808 01:48:37.056993 442 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: No bootstrap required, opened a new log
I20260808 01:48:37.057413 442 ts_tablet_manager.cc:1403] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.011s
I20260808 01:48:37.057783 444 consensus_queue.cc:260] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.055272 441 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632: No bootstrap required, opened a new log
I20260808 01:48:37.059652 441 ts_tablet_manager.cc:1403] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.007s
I20260808 01:48:37.068255 437 ts_tablet_manager.cc:1434] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent starting tablet: real 0.051s user 0.021s sys 0.001s
I20260808 01:48:37.072140 430 raft_consensus.cc:359] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.073163 430 raft_consensus.cc:385] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.073482 430 raft_consensus.cc:740] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.074496 430 consensus_queue.cc:260] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.079484 433 raft_consensus.cc:359] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.081504 433 raft_consensus.cc:385] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.081903 433 raft_consensus.cc:740] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.078666 436 raft_consensus.cc:359] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.088462 436 raft_consensus.cc:385] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.089715 436 raft_consensus.cc:740] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.080757 435 raft_consensus.cc:359] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.091468 435 raft_consensus.cc:385] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.091861 435 raft_consensus.cc:740] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.093339 431 ts_tablet_manager.cc:1434] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent starting tablet: real 0.107s user 0.026s sys 0.015s
I20260808 01:48:37.094444 435 consensus_queue.cc:260] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.082736 433 consensus_queue.cc:260] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.092970 436 consensus_queue.cc:260] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.098591 434 raft_consensus.cc:359] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.106549 432 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632: No bootstrap required, opened a new log
I20260808 01:48:37.106663 434 raft_consensus.cc:385] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.107056 432 ts_tablet_manager.cc:1403] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.007s
I20260808 01:48:37.107132 434 raft_consensus.cc:740] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.109488 434 consensus_queue.cc:260] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.111423 438 ts_tablet_manager.cc:1434] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent starting tablet: real 0.081s user 0.017s sys 0.004s
I20260808 01:48:37.117105 444 ts_tablet_manager.cc:1434] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent starting tablet: real 0.066s user 0.022s sys 0.005s
W20260808 01:48:37.121031 32592 tablet.cc:2406] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:37.122788 434 ts_tablet_manager.cc:1434] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.089s user 0.008s sys 0.007s
I20260808 01:48:37.114404 470 raft_consensus.cc:493] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.117074 436 ts_tablet_manager.cc:1434] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.125s user 0.010s sys 0.007s
I20260808 01:48:37.123672 464 raft_consensus.cc:493] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.124441 464 raft_consensus.cc:515] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.125541 470 raft_consensus.cc:515] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.119354 433 ts_tablet_manager.cc:1434] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.116s user 0.021s sys 0.000s
I20260808 01:48:37.121004 430 ts_tablet_manager.cc:1434] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.101s user 0.008s sys 0.005s
W20260808 01:48:37.130791 32733 tablet.cc:2406] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:37.139744 439 ts_tablet_manager.cc:1434] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Time spent starting tablet: real 0.106s user 0.034s sys 0.029s
I20260808 01:48:37.130937 435 ts_tablet_manager.cc:1434] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.074s user 0.014s sys 0.002s
I20260808 01:48:37.143553 442 raft_consensus.cc:359] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.144553 442 raft_consensus.cc:385] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.144876 442 raft_consensus.cc:740] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.146374 441 raft_consensus.cc:359] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.147172 441 raft_consensus.cc:385] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.147483 441 raft_consensus.cc:740] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.148577 470 leader_election.cc:290] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.148358 441 consensus_queue.cc:260] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.132826 464 leader_election.cc:290] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.149892 442 consensus_queue.cc:260] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.157536 443 raft_consensus.cc:359] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.159088 443 raft_consensus.cc:385] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.158951 440 raft_consensus.cc:359] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.159431 443 raft_consensus.cc:740] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.159709 440 raft_consensus.cc:385] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.160022 440 raft_consensus.cc:740] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.160759 440 consensus_queue.cc:260] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.160759 443 consensus_queue.cc:260] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.179919 440 ts_tablet_manager.cc:1434] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632: Time spent starting tablet: real 0.134s user 0.014s sys 0.007s
I20260808 01:48:37.181833 441 ts_tablet_manager.cc:1434] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632: Time spent starting tablet: real 0.122s user 0.008s sys 0.009s
I20260808 01:48:37.158281 432 raft_consensus.cc:359] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.183936 432 raft_consensus.cc:385] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:37.184247 432 raft_consensus.cc:740] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Initialized, Role: FOLLOWER
I20260808 01:48:37.184384 443 ts_tablet_manager.cc:1434] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Time spent starting tablet: real 0.129s user 0.017s sys 0.000s
I20260808 01:48:37.185323 432 consensus_queue.cc:260] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.190312 32545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f" is_pre_election: true
I20260808 01:48:37.190682 442 ts_tablet_manager.cc:1434] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Time spent starting tablet: real 0.133s user 0.009s sys 0.007s
I20260808 01:48:37.191263 32545 raft_consensus.cc:2468] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.191943 462 raft_consensus.cc:493] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.193095 32618 leader_election.cc:304] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.192705 462 raft_consensus.cc:515] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.194144 404 heartbeater.cc:499] Master 127.29.180.126:41303 was elected leader, sending a full tablet report...
I20260808 01:48:37.195531 462 leader_election.cc:290] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.196920 432 ts_tablet_manager.cc:1434] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632: Time spent starting tablet: real 0.090s user 0.015s sys 0.003s
I20260808 01:48:37.201015 32545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6482d8c7bc374c88826a3e3f295c2021" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f" is_pre_election: true
I20260808 01:48:37.201679 32545 raft_consensus.cc:2468] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.202832 462 raft_consensus.cc:2804] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.203230 462 raft_consensus.cc:493] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.203564 462 raft_consensus.cc:3060] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.207268 359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.208253 359 raft_consensus.cc:2468] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.209486 32618 leader_election.cc:304] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.211133 464 raft_consensus.cc:2804] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.210942 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6482d8c7bc374c88826a3e3f295c2021" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.211493 464 raft_consensus.cc:493] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.211630 358 raft_consensus.cc:2468] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.211828 464 raft_consensus.cc:3060] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.212930 359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.213469 359 raft_consensus.cc:2468] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 0.
I20260808 01:48:37.215497 32476 leader_election.cc:304] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.215790 462 raft_consensus.cc:515] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.216499 470 raft_consensus.cc:2804] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.216861 470 raft_consensus.cc:493] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.217204 470 raft_consensus.cc:3060] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.217777 462 leader_election.cc:290] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.218173 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" is_pre_election: true
I20260808 01:48:37.219244 32687 raft_consensus.cc:2393] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0702aa63d2a34898b13be46470a6dd1f in current term 1: Already voted for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in this term.
I20260808 01:48:37.219854 464 raft_consensus.cc:515] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.220461 359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.220981 359 raft_consensus.cc:3060] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.221876 464 leader_election.cc:290] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.221869 32545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f"
I20260808 01:48:37.223750 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6482d8c7bc374c88826a3e3f295c2021" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.223635 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6482d8c7bc374c88826a3e3f295c2021" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f"
I20260808 01:48:37.224330 32544 raft_consensus.cc:3060] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.224318 358 raft_consensus.cc:3060] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.224294 470 raft_consensus.cc:515] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.226944 470 leader_election.cc:290] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Requested vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.228412 359 raft_consensus.cc:2468] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 1.
I20260808 01:48:37.229043 32545 raft_consensus.cc:2393] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in current term 1: Already voted for candidate 0702aa63d2a34898b13be46470a6dd1f in this term.
I20260808 01:48:37.228976 357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.230074 357 raft_consensus.cc:2393] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0702aa63d2a34898b13be46470a6dd1f in current term 1: Already voted for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in this term.
I20260808 01:48:37.229976 32618 leader_election.cc:304] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.231256 464 raft_consensus.cc:2804] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:37.231266 358 raft_consensus.cc:2468] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 1.
I20260808 01:48:37.231945 464 raft_consensus.cc:697] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Becoming Leader. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Running, Role: LEADER
I20260808 01:48:37.232118 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1fcf8b4e5b1403b97bad94682add9e5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
I20260808 01:48:37.232546 32618 leader_election.cc:304] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.233280 462 raft_consensus.cc:2804] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:37.232975 464 consensus_queue.cc:237] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.233763 462 raft_consensus.cc:697] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Becoming Leader. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Running, Role: LEADER
I20260808 01:48:37.234405 32477 leader_election.cc:304] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f; no voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632
I20260808 01:48:37.234534 462 consensus_queue.cc:237] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.235603 470 raft_consensus.cc:2749] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:48:37.236984 32544 raft_consensus.cc:2468] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 1.
I20260808 01:48:37.247665 32390 catalog_manager.cc:5697] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:37.259099 32390 catalog_manager.cc:5697] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:37.276904 470 raft_consensus.cc:493] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.277361 470 raft_consensus.cc:515] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.279004 470 leader_election.cc:290] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.280025 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" is_pre_election: true
I20260808 01:48:37.280280 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.280558 32687 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 0.
I20260808 01:48:37.280869 358 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 0.
I20260808 01:48:37.281741 32477 leader_election.cc:304] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.282450 470 raft_consensus.cc:2804] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.282791 470 raft_consensus.cc:493] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.283068 470 raft_consensus.cc:3060] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.287415 470 raft_consensus.cc:515] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.289058 470 leader_election.cc:290] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Requested vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.289915 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
I20260808 01:48:37.290189 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.290418 32687 raft_consensus.cc:3060] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.290699 358 raft_consensus.cc:3060] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.295274 358 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 1.
I20260808 01:48:37.295303 32687 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 1.
I20260808 01:48:37.296386 32476 leader_election.cc:304] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.297163 470 raft_consensus.cc:2804] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:37.297580 470 raft_consensus.cc:697] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 1 LEADER]: Becoming Leader. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Running, Role: LEADER
I20260808 01:48:37.298511 470 consensus_queue.cc:237] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.308183 32390 catalog_manager.cc:5697] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65). New cstate: current_term: 1 leader_uuid: "0702aa63d2a34898b13be46470a6dd1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
W20260808 01:48:37.355654 405 tablet.cc:2406] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:37.457005 462 raft_consensus.cc:493] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.457479 462 raft_consensus.cc:515] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.459272 462 leader_election.cc:290] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.459966 470 raft_consensus.cc:493] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.460206 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f" is_pre_election: true
I20260808 01:48:37.460386 470 raft_consensus.cc:515] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.460605 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.461272 358 raft_consensus.cc:2468] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.461292 32544 raft_consensus.cc:2468] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.462850 470 leader_election.cc:290] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.463060 32618 leader_election.cc:304] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.463173 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" is_pre_election: true
I20260808 01:48:37.463321 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.463840 358 raft_consensus.cc:2468] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 0.
I20260808 01:48:37.463837 32687 raft_consensus.cc:2468] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0702aa63d2a34898b13be46470a6dd1f in term 0.
I20260808 01:48:37.464376 462 raft_consensus.cc:2804] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.464793 462 raft_consensus.cc:493] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.464908 32476 leader_election.cc:304] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.465124 462 raft_consensus.cc:3060] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.465754 470 raft_consensus.cc:2804] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.466183 470 raft_consensus.cc:493] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.466514 470 raft_consensus.cc:3060] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.472025 462 raft_consensus.cc:515] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.473044 470 raft_consensus.cc:515] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.474229 462 leader_election.cc:290] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.474920 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f"
I20260808 01:48:37.475349 470 leader_election.cc:290] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Requested vote from peers aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.475325 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.475699 32544 raft_consensus.cc:2393] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in current term 1: Already voted for candidate 0702aa63d2a34898b13be46470a6dd1f in this term.
I20260808 01:48:37.475891 358 raft_consensus.cc:3060] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.476353 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
I20260808 01:48:37.477084 32687 raft_consensus.cc:2393] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0702aa63d2a34898b13be46470a6dd1f in current term 1: Already voted for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in this term.
I20260808 01:48:37.477007 357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5181f3f52b649e2847c9b9107d3f7f5" candidate_uuid: "0702aa63d2a34898b13be46470a6dd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.478600 32476 leader_election.cc:304] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f; no voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632
I20260808 01:48:37.479486 470 raft_consensus.cc:2749] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:48:37.481520 358 raft_consensus.cc:2468] T c5181f3f52b649e2847c9b9107d3f7f5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 1.
I20260808 01:48:37.482589 32618 leader_election.cc:304] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632; no voters: 0702aa63d2a34898b13be46470a6dd1f
I20260808 01:48:37.483230 462 raft_consensus.cc:2804] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:37.483650 462 raft_consensus.cc:697] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Becoming Leader. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Running, Role: LEADER
I20260808 01:48:37.484316 462 consensus_queue.cc:237] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.491037 474 raft_consensus.cc:493] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.491460 474 raft_consensus.cc:515] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.492354 32390 catalog_manager.cc:5697] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:37.493783 462 raft_consensus.cc:493] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:37.494248 462 raft_consensus.cc:515] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.496984 462 leader_election.cc:290] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.499204 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f" is_pre_election: true
I20260808 01:48:37.499871 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
I20260808 01:48:37.499873 32544 raft_consensus.cc:2468] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.500528 358 raft_consensus.cc:2468] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 0.
I20260808 01:48:37.499886 474 leader_election.cc:290] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:37.501569 32618 leader_election.cc:304] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.505479 462 raft_consensus.cc:2804] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.505934 462 raft_consensus.cc:493] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.506279 462 raft_consensus.cc:3060] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.512970 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f" is_pre_election: true
I20260808 01:48:37.513058 462 raft_consensus.cc:515] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.513494 32544 raft_consensus.cc:2468] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bf8ae821ac41859a4aebb8ede00632 in term 0.
I20260808 01:48:37.515241 32759 leader_election.cc:304] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:37.516165 462 leader_election.cc:290] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:37.516500 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f"
I20260808 01:48:37.516717 474 raft_consensus.cc:2804] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:37.516767 358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632"
I20260808 01:48:37.517053 32544 raft_consensus.cc:3060] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.517155 474 raft_consensus.cc:493] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:37.517493 474 raft_consensus.cc:3060] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:37.519942 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" is_pre_election: true
I20260808 01:48:37.520556 32687 raft_consensus.cc:2393] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6bf8ae821ac41859a4aebb8ede00632 in current term 1: Already voted for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in this term.
I20260808 01:48:37.524009 32544 raft_consensus.cc:2468] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in term 1.
I20260808 01:48:37.524632 474 raft_consensus.cc:515] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.525151 32618 leader_election.cc:304] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters:
I20260808 01:48:37.525741 358 raft_consensus.cc:2393] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in current term 1: Already voted for candidate c6bf8ae821ac41859a4aebb8ede00632 in this term.
I20260808 01:48:37.525997 462 raft_consensus.cc:2804] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:37.526677 474 leader_election.cc:290] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 1 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:37.527616 32544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0702aa63d2a34898b13be46470a6dd1f"
I20260808 01:48:37.528344 32544 raft_consensus.cc:2393] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6bf8ae821ac41859a4aebb8ede00632 in current term 1: Already voted for candidate aaaed6a8a7ad4c0389a4daad6b2b4dc4 in this term.
I20260808 01:48:37.528537 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e319d6c4dac04f1b9856214abe251b74" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
I20260808 01:48:37.529348 462 raft_consensus.cc:697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Becoming Leader. State: Replica: aaaed6a8a7ad4c0389a4daad6b2b4dc4, State: Running, Role: LEADER
I20260808 01:48:37.530216 462 consensus_queue.cc:237] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:37.536209 32760 leader_election.cc:304] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6bf8ae821ac41859a4aebb8ede00632; no voters: 0702aa63d2a34898b13be46470a6dd1f, aaaed6a8a7ad4c0389a4daad6b2b4dc4
I20260808 01:48:37.537046 474 raft_consensus.cc:2749] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:48:37.538080 32391 catalog_manager.cc:5697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:37.636780 462 consensus_queue.cc:1048] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:37.683750 470 consensus_queue.cc:1048] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:37.686663 462 consensus_queue.cc:1048] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:37.714365 462 consensus_queue.cc:1048] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:37.775651 468 consensus_queue.cc:1048] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:37.797338 462 consensus_queue.cc:1048] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:37.856355 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 32453
I20260808 01:48:37.895138 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:34975
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:41303
--webserver_port=39497
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:48:37.918722 464 consensus_queue.cc:1048] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:37.921785 464 consensus_queue.cc:1048] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:37.940001 32618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111)
W20260808 01:48:37.954458 32618 consensus_peers.cc:597] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:37.958243 32618 consensus_peers.cc:597] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:38.127274 32618 consensus_peers.cc:597] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:38.245427 504 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:38.246042 504 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:38.246719 504 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:38.282845 504 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:48:38.283329 504 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:38.284642 504 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:38.323132 504 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:34975
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=39497
--tserver_master_addrs=127.29.180.126:41303
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:38.324676 504 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:48:38.325659 504 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:48:38.325832 504 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:48:38.326957 504 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:38.340541 515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:38.363631 32618 consensus_peers.cc:597] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:39.243213 508 raft_consensus.cc:493] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0702aa63d2a34898b13be46470a6dd1f)
I20260808 01:48:39.244154 508 raft_consensus.cc:515] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:39.247781 508 leader_election.cc:290] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:39.253561 359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c6bf8ae821ac41859a4aebb8ede00632" is_pre_election: true
W20260808 01:48:39.261658 32618 leader_election.cc:336] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111)
I20260808 01:48:39.262516 32618 leader_election.cc:304] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4; no voters: 0702aa63d2a34898b13be46470a6dd1f, c6bf8ae821ac41859a4aebb8ede00632
I20260808 01:48:39.263829 508 raft_consensus.cc:2749] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:48:39.297506 522 raft_consensus.cc:493] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0702aa63d2a34898b13be46470a6dd1f)
I20260808 01:48:39.303560 522 raft_consensus.cc:515] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
W20260808 01:48:39.313319 32759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111)
I20260808 01:48:39.314926 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" is_pre_election: true
I20260808 01:48:39.315856 32687 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bf8ae821ac41859a4aebb8ede00632 in term 1.
I20260808 01:48:39.318327 32760 leader_election.cc:304] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632; no voters:
I20260808 01:48:39.323397 522 leader_election.cc:290] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:39.325331 522 raft_consensus.cc:2804] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:48:39.326361 522 raft_consensus.cc:493] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0702aa63d2a34898b13be46470a6dd1f)
I20260808 01:48:39.329146 522 raft_consensus.cc:3060] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:48:39.341033 32759 leader_election.cc:336] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111)
I20260808 01:48:39.346786 522 raft_consensus.cc:515] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:39.351984 32687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef18b6b84f7b40419d7f568744c17b24" candidate_uuid: "c6bf8ae821ac41859a4aebb8ede00632" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4"
I20260808 01:48:39.352901 32687 raft_consensus.cc:3060] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:48:39.359124 32759 leader_election.cc:336] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111)
I20260808 01:48:39.364287 522 leader_election.cc:290] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 election: Requested vote from peers 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975), aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:39.367422 32687 raft_consensus.cc:2468] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bf8ae821ac41859a4aebb8ede00632 in term 2.
I20260808 01:48:39.369578 32760 leader_election.cc:304] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aaaed6a8a7ad4c0389a4daad6b2b4dc4, c6bf8ae821ac41859a4aebb8ede00632; no voters: 0702aa63d2a34898b13be46470a6dd1f
I20260808 01:48:39.374460 522 raft_consensus.cc:2804] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:39.375526 522 raft_consensus.cc:697] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 2 LEADER]: Becoming Leader. State: Replica: c6bf8ae821ac41859a4aebb8ede00632, State: Running, Role: LEADER
I20260808 01:48:39.376787 522 consensus_queue.cc:237] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:39.400321 32389 catalog_manager.cc:5697] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 reported cstate change: term changed from 1 to 2, leader changed from 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) to c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67). New cstate: current_term: 2 leader_uuid: "c6bf8ae821ac41859a4aebb8ede00632" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
W20260808 01:48:38.340960 516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:39.794502 504 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.454s user 0.438s sys 0.839s
W20260808 01:48:39.794930 504 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.454s user 0.438s sys 0.839s
W20260808 01:48:39.745460 514 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 504
W20260808 01:48:39.796990 518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:39.800493 517 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1454 milliseconds
I20260808 01:48:39.800557 504 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:39.802176 504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:39.804523 504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:39.805974 504 hybrid_clock.cc:648] HybridClock initialized: now 1786153719805855 us; error 66 us; skew 500 ppm
I20260808 01:48:39.807003 504 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:39.813791 504 webserver.cc:533] Webserver started at http://127.29.180.65:39497/ using document root <none> and password file <none>
I20260808 01:48:39.815683 504 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:39.815999 504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:48:39.818670 504 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:39.822341 32687 raft_consensus.cc:1275] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Refusing update from remote peer c6bf8ae821ac41859a4aebb8ede00632: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:39.823853 522 consensus_queue.cc:1048] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
E20260808 01:48:39.824863 504 env_posix.cc:431] I/O error, context: INJECTED FAILURE
W20260808 01:48:39.844158 32759 consensus_peers.cc:597] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
E20260808 01:48:39.849066 504 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:39.861188 504 fs_manager.cc:714] Time spent opening directory manager: real 0.039s user 0.033s sys 0.002s
E20260808 01:48:39.867257 504 log_block_manager.cc:3241] Not using report from /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260808 01:48:39.867605 543 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:39.868088 543 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:39.869704 543 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:39.869831 544 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:39.871479 504 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260808 01:48:39.871852 504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "0702aa63d2a34898b13be46470a6dd1f"
format_stamp: "Formatted at 2026-08-08 01:48:31 on dist-test-slave-p3t5"
I20260808 01:48:39.872524 504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:39.940708 504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:39.942436 504 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:39.942963 504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:39.945895 504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:39.952931 551 ts_tablet_manager.cc:548] Loading tablet metadata (0/5 complete)
W20260808 01:48:39.966435 552 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet e319d6c4dac04f1b9856214abe251b74: could not find data dir with uuid 326290ccbe3546c79297898f846fec36
W20260808 01:48:39.969796 553 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5: could not find data dir with uuid 326290ccbe3546c79297898f846fec36
W20260808 01:48:39.973024 551 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 6482d8c7bc374c88826a3e3f295c2021: could not find data dir with uuid 326290ccbe3546c79297898f846fec36
W20260808 01:48:39.976357 554 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet ef18b6b84f7b40419d7f568744c17b24: could not find data dir with uuid 326290ccbe3546c79297898f846fec36
I20260808 01:48:39.976907 504 ts_tablet_manager.cc:585] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260808 01:48:39.977133 504 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.026s user 0.015s sys 0.005s
I20260808 01:48:39.977465 504 ts_tablet_manager.cc:600] Registering tablets (0/5 complete)
I20260808 01:48:39.983718 554 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:39.987453 551 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
E20260808 01:48:39.988363 551 ts_tablet_manager.cc:1425] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet e319d6c4dac04f1b9856214abe251b74
I20260808 01:48:39.988765 551 ts_tablet_manager.cc:1403] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 01:48:39.989156 551 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:39.991104 552 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
E20260808 01:48:39.991622 552 ts_tablet_manager.cc:1425] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5
I20260808 01:48:39.992004 552 ts_tablet_manager.cc:1403] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 01:48:39.992316 552 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:39.992599 552 raft_consensus.cc:2243] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:39.989571 551 raft_consensus.cc:2243] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:39.993170 552 raft_consensus.cc:2272] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:39.993286 551 raft_consensus.cc:2272] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:39.996138 551 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
E20260808 01:48:39.996610 551 ts_tablet_manager.cc:1425] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 6482d8c7bc374c88826a3e3f295c2021
I20260808 01:48:39.996923 551 ts_tablet_manager.cc:1403] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 01:48:39.997193 551 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:39.997438 551 raft_consensus.cc:2243] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:39.997774 551 raft_consensus.cc:2272] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:40.001151 504 ts_tablet_manager.cc:616] Registered 5 tablets
I20260808 01:48:40.001502 504 ts_tablet_manager.cc:595] Time spent register tablets: real 0.024s user 0.021s sys 0.000s
I20260808 01:48:40.001546 551 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
E20260808 01:48:40.002095 551 ts_tablet_manager.cc:1425] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:40.002494 551 ts_tablet_manager.cc:1403] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 01:48:40.002869 551 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:40.003216 551 raft_consensus.cc:2243] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:40.003722 551 raft_consensus.cc:2272] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:40.037484 554 log.cc:826] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:40.044675 554 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:40.046139 554 tablet_bootstrap.cc:492] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap complete.
I20260808 01:48:40.048555 554 ts_tablet_manager.cc:1403] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.065s user 0.039s sys 0.023s
I20260808 01:48:40.079582 554 raft_consensus.cc:359] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.081063 554 raft_consensus.cc:740] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:40.082314 554 consensus_queue.cc:260] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.087230 554 ts_tablet_manager.cc:1434] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.038s user 0.040s sys 0.000s
W20260808 01:48:40.111076 32618 consensus_peers.cc:597] T c5181f3f52b649e2847c9b9107d3f7f5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:34975: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:40.172539 504 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:34975
I20260808 01:48:40.172780 664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:34975 every 8 connection(s)
I20260808 01:48:40.175632 504 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
W20260808 01:48:40.181910 667 tablet.cc:2406] T c5181f3f52b649e2847c9b9107d3f7f5 P 0702aa63d2a34898b13be46470a6dd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:40.186152 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 504
I20260808 01:48:40.202155 665 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41303
I20260808 01:48:40.202706 665 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:40.204103 508 raft_consensus.cc:1012] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Attempting to remove follower 0702aa63d2a34898b13be46470a6dd1f from the Raft config. Reason: The tablet replica hosted on peer 0702aa63d2a34898b13be46470a6dd1f has failed
I20260808 01:48:40.204144 665 heartbeater.cc:507] Master 127.29.180.126:41303 requested a full tablet report, sending...
I20260808 01:48:40.206089 508 consensus_queue.cc:237] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.210361 32389 ts_manager.cc:194] Re-registered known tserver with Master: 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975)
I20260808 01:48:40.211699 359 raft_consensus.cc:1275] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:40.213370 671 consensus_queue.cc:1048] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:48:40.213798 32389 catalog_manager.cc:5610] Tablet b1fcf8b4e5b1403b97bad94682add9e5 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5
W20260808 01:48:40.214214 32389 catalog_manager.cc:5610] Tablet e319d6c4dac04f1b9856214abe251b74 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet e319d6c4dac04f1b9856214abe251b74
W20260808 01:48:40.214474 32389 catalog_manager.cc:5610] Tablet 6482d8c7bc374c88826a3e3f295c2021 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 6482d8c7bc374c88826a3e3f295c2021
W20260808 01:48:40.214728 32389 catalog_manager.cc:5610] Tablet ef18b6b84f7b40419d7f568744c17b24 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:40.218700 32389 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:51229
I20260808 01:48:40.221699 508 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.223836 359 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.235383 32389 catalog_manager.cc:5697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.253700 32687 consensus_queue.cc:237] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:40.261960 359 raft_consensus.cc:1275] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:48:40.262104 32618 consensus_peers.cc:597] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet e319d6c4dac04f1b9856214abe251b74. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:40.263702 671 consensus_queue.cc:1048] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:40.265415 598 tablet_service.cc:1558] Processing DeleteTablet for tablet e319d6c4dac04f1b9856214abe251b74 with delete_type TABLET_DATA_TOMBSTONED (TS 0702aa63d2a34898b13be46470a6dd1f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50694
I20260808 01:48:40.268992 684 ts_tablet_manager.cc:1916] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:40.271952 671 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.277415 359 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.282935 32376 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet e319d6c4dac04f1b9856214abe251b74 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:40.289922 684 ts_tablet_manager.cc:1929] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:40.290335 684 log.cc:1199] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/e319d6c4dac04f1b9856214abe251b74
I20260808 01:48:40.289559 32389 catalog_manager.cc:5697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.292970 32375 catalog_manager.cc:5028] TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): tablet e319d6c4dac04f1b9856214abe251b74 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) successfully deleted
I20260808 01:48:40.321141 526 raft_consensus.cc:1012] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Attempting to remove follower 0702aa63d2a34898b13be46470a6dd1f from the Raft config. Reason: The tablet replica hosted on peer 0702aa63d2a34898b13be46470a6dd1f has failed
I20260808 01:48:40.322962 526 consensus_queue.cc:237] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.327219 32687 raft_consensus.cc:1275] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Refusing update from remote peer c6bf8ae821ac41859a4aebb8ede00632: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:48:40.328732 538 consensus_queue.cc:1048] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:40.335392 526 raft_consensus.cc:2955] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.337082 32687 raft_consensus.cc:2955] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.349706 32391 catalog_manager.cc:5697] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index -1 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "c6bf8ae821ac41859a4aebb8ede00632" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.359565 598 tablet_service.cc:1558] Processing DeleteTablet for tablet ef18b6b84f7b40419d7f568744c17b24 with delete_type TABLET_DATA_TOMBSTONED (TS 0702aa63d2a34898b13be46470a6dd1f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50694
I20260808 01:48:40.360838 684 ts_tablet_manager.cc:1916] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:40.363013 359 consensus_queue.cc:237] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
W20260808 01:48:40.369592 32759 consensus_peers.cc:597] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:40.369786 32687 raft_consensus.cc:1275] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Refusing update from remote peer c6bf8ae821ac41859a4aebb8ede00632: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:48:40.371206 526 consensus_queue.cc:1048] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:40.378216 684 ts_tablet_manager.cc:1929] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:40.378611 684 log.cc:1199] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:40.378135 538 raft_consensus.cc:2955] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.380638 32375 catalog_manager.cc:5028] TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): tablet ef18b6b84f7b40419d7f568744c17b24 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) successfully deleted
I20260808 01:48:40.382248 32687 raft_consensus.cc:2955] T ef18b6b84f7b40419d7f568744c17b24 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.387246 32375 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet ef18b6b84f7b40419d7f568744c17b24 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:40.392622 32391 catalog_manager.cc:5697] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632 reported cstate change: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New cstate: current_term: 2 leader_uuid: "c6bf8ae821ac41859a4aebb8ede00632" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.583226 598 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:40.586141 671 raft_consensus.cc:1012] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Attempting to remove follower 0702aa63d2a34898b13be46470a6dd1f from the Raft config. Reason: The tablet replica hosted on peer 0702aa63d2a34898b13be46470a6dd1f has failed
W20260808 01:48:40.586058 32618 consensus_peers.cc:597] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:40.587988 671 consensus_queue.cc:237] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.588815 338 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:48:40.589129 32666 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:40.596144 359 raft_consensus.cc:1275] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:40.597945 508 consensus_queue.cc:1048] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:40.619642 508 raft_consensus.cc:2955] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.623047 357 raft_consensus.cc:2955] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.636958 32389 catalog_manager.cc:5697] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 reported cstate change: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.649055 598 tablet_service.cc:1558] Processing DeleteTablet for tablet b1fcf8b4e5b1403b97bad94682add9e5 with delete_type TABLET_DATA_TOMBSTONED (TS 0702aa63d2a34898b13be46470a6dd1f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50694
I20260808 01:48:40.650017 684 ts_tablet_manager.cc:1916] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:40.671515 684 ts_tablet_manager.cc:1929] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:40.672015 684 log.cc:1199] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/b1fcf8b4e5b1403b97bad94682add9e5
I20260808 01:48:40.674276 32375 catalog_manager.cc:5028] TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): tablet b1fcf8b4e5b1403b97bad94682add9e5 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) successfully deleted
I20260808 01:48:40.676690 32687 consensus_queue.cc:237] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:40.696187 357 raft_consensus.cc:1275] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:48:40.700492 671 consensus_queue.cc:1048] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:48:40.702669 32618 consensus_peers.cc:597] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:40.705440 671 raft_consensus.cc:1012] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Attempting to remove follower 0702aa63d2a34898b13be46470a6dd1f from the Raft config. Reason: The tablet replica hosted on peer 0702aa63d2a34898b13be46470a6dd1f has failed
I20260808 01:48:40.707527 671 consensus_queue.cc:237] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.728694 357 raft_consensus.cc:1275] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:48:40.728153 707 raft_consensus.cc:2955] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.739059 32376 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet b1fcf8b4e5b1403b97bad94682add9e5 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:40.740818 665 heartbeater.cc:499] Master 127.29.180.126:41303 was elected leader, sending a full tablet report...
I20260808 01:48:40.744735 32389 catalog_manager.cc:5697] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.757663 357 raft_consensus.cc:2955] T b1fcf8b4e5b1403b97bad94682add9e5 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.771273 671 consensus_queue.cc:1048] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:40.791049 359 raft_consensus.cc:2955] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.788939 671 raft_consensus.cc:2955] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.805267 32389 catalog_manager.cc:5697] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 reported cstate change: config changed from index -1 to 2, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.806010 712 ts_tablet_manager.cc:933] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Initiating tablet copy from peer c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183)
I20260808 01:48:40.816773 598 tablet_service.cc:1558] Processing DeleteTablet for tablet 6482d8c7bc374c88826a3e3f295c2021 with delete_type TABLET_DATA_TOMBSTONED (TS 0702aa63d2a34898b13be46470a6dd1f not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50694
I20260808 01:48:40.817775 684 ts_tablet_manager.cc:1916] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:48:40.830859 712 tablet_copy_client.cc:286] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:48:40.833000 32687 consensus_queue.cc:237] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:40.831552 712 tablet_copy_client.cc:323] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:33183
W20260808 01:48:40.843598 32618 consensus_peers.cc:597] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:40.846774 357 raft_consensus.cc:1275] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:48:40.851405 671 consensus_queue.cc:1048] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:40.858551 707 raft_consensus.cc:2955] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.862103 684 ts_tablet_manager.cc:1929] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:48:40.862526 684 log.cc:1199] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/6482d8c7bc374c88826a3e3f295c2021
I20260808 01:48:40.870067 357 raft_consensus.cc:2955] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.886077 716 raft_consensus.cc:1012] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Attempting to remove follower 0702aa63d2a34898b13be46470a6dd1f from the Raft config. Reason: The tablet replica hosted on peer 0702aa63d2a34898b13be46470a6dd1f has failed
I20260808 01:48:40.887966 716 consensus_queue.cc:237] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } }
I20260808 01:48:40.887630 32389 catalog_manager.cc:5697] T 6482d8c7bc374c88826a3e3f295c2021 P c6bf8ae821ac41859a4aebb8ede00632 reported cstate change: config changed from index 2 to 3, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.906134 32376 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6482d8c7bc374c88826a3e3f295c2021 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20260808 01:48:40.907353 32391 catalog_manager.cc:5610] Tablet b1fcf8b4e5b1403b97bad94682add9e5 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5
W20260808 01:48:40.907811 32391 catalog_manager.cc:5610] Tablet e319d6c4dac04f1b9856214abe251b74 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet e319d6c4dac04f1b9856214abe251b74
W20260808 01:48:40.908365 32391 catalog_manager.cc:5610] Tablet 6482d8c7bc374c88826a3e3f295c2021 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 6482d8c7bc374c88826a3e3f295c2021
W20260808 01:48:40.908727 32391 catalog_manager.cc:5610] Tablet ef18b6b84f7b40419d7f568744c17b24 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) has failed on TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:40.923959 357 raft_consensus.cc:1275] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:48:40.929417 32375 catalog_manager.cc:5028] TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): tablet 6482d8c7bc374c88826a3e3f295c2021 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) successfully deleted
I20260808 01:48:40.929569 718 consensus_queue.cc:1048] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:40.936956 508 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.939270 357 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.944519 379 tablet_copy_service.cc:140] P c6bf8ae821ac41859a4aebb8ede00632: Received BeginTabletCopySession request for tablet ef18b6b84f7b40419d7f568744c17b24 from peer 0702aa63d2a34898b13be46470a6dd1f ({username='slave'} at 127.29.180.65:49915)
I20260808 01:48:40.945174 379 tablet_copy_service.cc:161] P c6bf8ae821ac41859a4aebb8ede00632: Beginning new tablet copy session on tablet ef18b6b84f7b40419d7f568744c17b24 from peer 0702aa63d2a34898b13be46470a6dd1f at {username='slave'} at 127.29.180.65:49915: session id = 0702aa63d2a34898b13be46470a6dd1f-ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:40.950631 32389 catalog_manager.cc:5697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index 3 to 4, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } }
I20260808 01:48:40.953168 379 tablet_copy_source_session.cc:215] T ef18b6b84f7b40419d7f568744c17b24 P c6bf8ae821ac41859a4aebb8ede00632: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:40.959499 712 ts_tablet_manager.cc:1916] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:40.962754 598 tablet_service.cc:1558] Processing DeleteTablet for tablet e319d6c4dac04f1b9856214abe251b74 with delete_type TABLET_DATA_TOMBSTONED (TS 0702aa63d2a34898b13be46470a6dd1f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50694
I20260808 01:48:40.965520 32375 catalog_manager.cc:5028] TS 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): tablet e319d6c4dac04f1b9856214abe251b74 (table test-workload [id=cccf5de361c347b692a5096b0b0b56b4]) successfully deleted
I20260808 01:48:40.966059 32687 consensus_queue.cc:237] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
W20260808 01:48:40.972364 32618 consensus_peers.cc:597] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 -> Peer 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): Couldn't send request to peer 0702aa63d2a34898b13be46470a6dd1f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:40.972571 357 raft_consensus.cc:1275] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Refusing update from remote peer aaaed6a8a7ad4c0389a4daad6b2b4dc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 01:48:40.973904 718 consensus_queue.cc:1048] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:48:40.979557 508 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.985394 357 raft_consensus.cc:2955] T e319d6c4dac04f1b9856214abe251b74 P c6bf8ae821ac41859a4aebb8ede00632 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.987509 712 ts_tablet_manager.cc:1929] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:40.988121 32376 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet e319d6c4dac04f1b9856214abe251b74 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:48:40.994334 32389 catalog_manager.cc:5697] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4 reported cstate change: config changed from index 4 to 5, VOTER 0702aa63d2a34898b13be46470a6dd1f (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } } }
I20260808 01:48:40.998490 712 tablet_copy_client.cc:806] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:40.999212 712 tablet_copy_client.cc:670] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:41.005553 712 tablet_copy_client.cc:538] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:41.016189 712 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:41.046844 712 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:41.048007 712 tablet_bootstrap.cc:492] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap complete.
I20260808 01:48:41.048725 712 ts_tablet_manager.cc:1403] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20260808 01:48:41.051625 712 raft_consensus.cc:359] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.052321 712 raft_consensus.cc:740] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:41.053047 712 consensus_queue.cc:260] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.055424 712 ts_tablet_manager.cc:1434] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
I20260808 01:48:41.057293 379 tablet_copy_service.cc:342] P c6bf8ae821ac41859a4aebb8ede00632: Request end of tablet copy session 0702aa63d2a34898b13be46470a6dd1f-ef18b6b84f7b40419d7f568744c17b24 received from {username='slave'} at 127.29.180.65:49915
I20260808 01:48:41.057757 379 tablet_copy_service.cc:434] P c6bf8ae821ac41859a4aebb8ede00632: ending tablet copy session 0702aa63d2a34898b13be46470a6dd1f-ef18b6b84f7b40419d7f568744c17b24 on tablet ef18b6b84f7b40419d7f568744c17b24 with peer 0702aa63d2a34898b13be46470a6dd1f
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
89c7916fdcd048d58b330891aa1c95db | 127.29.180.126:41303 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0702aa63d2a34898b13be46470a6dd1f | 127.29.180.65:34975 | HEALTHY | <none> | 0 | 0
aaaed6a8a7ad4c0389a4daad6b2b4dc4 | 127.29.180.66:40163 | HEALTHY | <none> | 4 | 0
c6bf8ae821ac41859a4aebb8ede00632 | 127.29.180.67:33183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:34975
env_inject_eio | 1 | hidden | 127.29.180.65:34975
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:34975
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:34975
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:40163
I20260808 01:48:41.095018 712 ts_tablet_manager.cc:933] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Initiating tablet copy from peer aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:33183
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:34975
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:40163
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:33183
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet b1fcf8b4e5b1403b97bad94682add9e5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): not running
I20260808 01:48:41.096632 712 tablet_copy_client.cc:286] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b1fcf8b4e5b1403b97bad94682add9e5
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
I20260808 01:48:41.097067 712 tablet_copy_client.cc:323] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:40163
All reported replicas are:
A = aaaed6a8a7ad4c0389a4daad6b2b4dc4
B = 0702aa63d2a34898b13be46470a6dd1f
C = c6bf8ae821ac41859a4aebb8ede00632
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* C | 1 | 2 | No
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c5181f3f52b649e2847c9b9107d3f7f5 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): RUNNING
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
All reported replicas are:
A = 0702aa63d2a34898b13be46470a6dd1f
B = aaaed6a8a7ad4c0389a4daad6b2b4dc4
C = c6bf8ae821ac41859a4aebb8ede00632
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet ef18b6b84f7b40419d7f568744c17b24 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING [LEADER]
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = aaaed6a8a7ad4c0389a4daad6b2b4dc4
B = c6bf8ae821ac41859a4aebb8ede00632
C = 0702aa63d2a34898b13be46470a6dd1f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 4 | Yes
B | A B* C | 2 | 4 | Yes
C | [config not available] | | |
Tablet e319d6c4dac04f1b9856214abe251b74 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = aaaed6a8a7ad4c0389a4daad6b2b4dc4
B = c6bf8ae821ac41859a4aebb8ede00632
C = 0702aa63d2a34898b13be46470a6dd1f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
Tablet 6482d8c7bc374c88826a3e3f295c2021 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 6482d8c7bc374c88826a3e3f295c2021
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
All reported replicas are:
I20260808 01:48:41.106858 32707 tablet_copy_service.cc:140] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Received BeginTabletCopySession request for tablet b1fcf8b4e5b1403b97bad94682add9e5 from peer 0702aa63d2a34898b13be46470a6dd1f ({username='slave'} at 127.29.180.65:56185)
A = 0702aa63d2a34898b13be46470a6dd1f
B = aaaed6a8a7ad4c0389a4daad6b2b4dc4
C = c6bf8ae821ac41859a4aebb8ede00632
I20260808 01:48:41.107440 32707 tablet_copy_service.cc:161] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Beginning new tablet copy session on tablet b1fcf8b4e5b1403b97bad94682add9e5 from peer 0702aa63d2a34898b13be46470a6dd1f at {username='slave'} at 127.29.180.65:56185: session id = 0702aa63d2a34898b13be46470a6dd1f-b1fcf8b4e5b1403b97bad94682add9e5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 0 | 0 | 4 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:48:41.114104 32707 tablet_copy_source_session.cc:215] T b1fcf8b4e5b1403b97bad94682add9e5 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:41.116751 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:48:41.117372 712 ts_tablet_manager.cc:1916] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:41.129335 712 ts_tablet_manager.cc:1929] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:41.135360 712 tablet_copy_client.cc:806] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:41.135993 712 tablet_copy_client.cc:670] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:41.139935 712 tablet_copy_client.cc:538] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:41.146428 712 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:41.163064 712 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:41.163866 712 tablet_bootstrap.cc:492] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap complete.
I20260808 01:48:41.164450 712 ts_tablet_manager.cc:1403] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260808 01:48:41.166226 712 raft_consensus.cc:359] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.166658 712 raft_consensus.cc:740] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:41.167117 712 consensus_queue.cc:260] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.168629 712 ts_tablet_manager.cc:1434] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 01:48:41.170449 32707 tablet_copy_service.cc:342] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Request end of tablet copy session 0702aa63d2a34898b13be46470a6dd1f-b1fcf8b4e5b1403b97bad94682add9e5 received from {username='slave'} at 127.29.180.65:56185
I20260808 01:48:41.170823 32707 tablet_copy_service.cc:434] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: ending tablet copy session 0702aa63d2a34898b13be46470a6dd1f-b1fcf8b4e5b1403b97bad94682add9e5 on tablet b1fcf8b4e5b1403b97bad94682add9e5 with peer 0702aa63d2a34898b13be46470a6dd1f
E20260808 01:48:41.237306 661 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:41.257989 712 ts_tablet_manager.cc:933] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Initiating tablet copy from peer aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:41.259078 712 tablet_copy_client.cc:286] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:48:41.259529 712 tablet_copy_client.cc:323] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:40163
I20260808 01:48:41.260927 32707 tablet_copy_service.cc:140] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Received BeginTabletCopySession request for tablet 6482d8c7bc374c88826a3e3f295c2021 from peer 0702aa63d2a34898b13be46470a6dd1f ({username='slave'} at 127.29.180.65:56185)
I20260808 01:48:41.261361 32707 tablet_copy_service.cc:161] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Beginning new tablet copy session on tablet 6482d8c7bc374c88826a3e3f295c2021 from peer 0702aa63d2a34898b13be46470a6dd1f at {username='slave'} at 127.29.180.65:56185: session id = 0702aa63d2a34898b13be46470a6dd1f-6482d8c7bc374c88826a3e3f295c2021
I20260808 01:48:41.266182 32707 tablet_copy_source_session.cc:215] T 6482d8c7bc374c88826a3e3f295c2021 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:41.268343 712 ts_tablet_manager.cc:1916] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:41.279690 712 ts_tablet_manager.cc:1929] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:41.285518 712 tablet_copy_client.cc:806] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:41.286265 712 tablet_copy_client.cc:670] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:41.290422 712 tablet_copy_client.cc:538] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:41.296162 712 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:41.313386 712 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:41.314253 712 tablet_bootstrap.cc:492] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap complete.
I20260808 01:48:41.314795 712 ts_tablet_manager.cc:1403] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20260808 01:48:41.316560 712 raft_consensus.cc:359] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.316998 712 raft_consensus.cc:740] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:41.317662 712 consensus_queue.cc:260] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.319239 712 ts_tablet_manager.cc:1434] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260808 01:48:41.321120 32707 tablet_copy_service.cc:342] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Request end of tablet copy session 0702aa63d2a34898b13be46470a6dd1f-6482d8c7bc374c88826a3e3f295c2021 received from {username='slave'} at 127.29.180.65:56185
I20260808 01:48:41.321516 32707 tablet_copy_service.cc:434] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: ending tablet copy session 0702aa63d2a34898b13be46470a6dd1f-6482d8c7bc374c88826a3e3f295c2021 on tablet 6482d8c7bc374c88826a3e3f295c2021 with peer 0702aa63d2a34898b13be46470a6dd1f
I20260808 01:48:41.323953 619 raft_consensus.cc:1217] T ef18b6b84f7b40419d7f568744c17b24 P 0702aa63d2a34898b13be46470a6dd1f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260808 01:48:41.397193 712 ts_tablet_manager.cc:933] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Initiating tablet copy from peer aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163)
I20260808 01:48:41.398518 712 tablet_copy_client.cc:286] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:48:41.399046 712 tablet_copy_client.cc:323] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:40163
I20260808 01:48:41.400610 32707 tablet_copy_service.cc:140] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Received BeginTabletCopySession request for tablet e319d6c4dac04f1b9856214abe251b74 from peer 0702aa63d2a34898b13be46470a6dd1f ({username='slave'} at 127.29.180.65:56185)
I20260808 01:48:41.401099 32707 tablet_copy_service.cc:161] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Beginning new tablet copy session on tablet e319d6c4dac04f1b9856214abe251b74 from peer 0702aa63d2a34898b13be46470a6dd1f at {username='slave'} at 127.29.180.65:56185: session id = 0702aa63d2a34898b13be46470a6dd1f-e319d6c4dac04f1b9856214abe251b74
I20260808 01:48:41.407533 32707 tablet_copy_source_session.cc:215] T e319d6c4dac04f1b9856214abe251b74 P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:48:41.409781 712 ts_tablet_manager.cc:1916] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:48:41.424819 712 ts_tablet_manager.cc:1929] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:48:41.432510 712 tablet_copy_client.cc:806] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 0 data blocks...
I20260808 01:48:41.433140 712 tablet_copy_client.cc:670] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Starting download of 1 WAL segments...
I20260808 01:48:41.436908 712 tablet_copy_client.cc:538] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:48:41.444880 712 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap starting.
I20260808 01:48:41.471732 712 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:41.472785 712 tablet_bootstrap.cc:492] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Bootstrap complete.
I20260808 01:48:41.473317 712 ts_tablet_manager.cc:1403] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Time spent bootstrapping tablet: real 0.029s user 0.029s sys 0.000s
I20260808 01:48:41.475881 712 raft_consensus.cc:359] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.476368 712 raft_consensus.cc:740] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702aa63d2a34898b13be46470a6dd1f, State: Initialized, Role: FOLLOWER
I20260808 01:48:41.476910 712 consensus_queue.cc:260] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aaaed6a8a7ad4c0389a4daad6b2b4dc4" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 40163 } } peers { permanent_uuid: "c6bf8ae821ac41859a4aebb8ede00632" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 33183 } } peers { permanent_uuid: "0702aa63d2a34898b13be46470a6dd1f" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 34975 } }
I20260808 01:48:41.479470 712 ts_tablet_manager.cc:1434] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 01:48:41.481511 32707 tablet_copy_service.cc:342] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: Request end of tablet copy session 0702aa63d2a34898b13be46470a6dd1f-e319d6c4dac04f1b9856214abe251b74 received from {username='slave'} at 127.29.180.65:56185
I20260808 01:48:41.481921 32707 tablet_copy_service.cc:434] P aaaed6a8a7ad4c0389a4daad6b2b4dc4: ending tablet copy session 0702aa63d2a34898b13be46470a6dd1f-e319d6c4dac04f1b9856214abe251b74 on tablet e319d6c4dac04f1b9856214abe251b74 with peer 0702aa63d2a34898b13be46470a6dd1f
I20260808 01:48:41.506989 619 raft_consensus.cc:1217] T b1fcf8b4e5b1403b97bad94682add9e5 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260808 01:48:41.572139 598 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:41.572196 32666 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:41.576220 338 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:48:41.772753 619 raft_consensus.cc:1217] T 6482d8c7bc374c88826a3e3f295c2021 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
89c7916fdcd048d58b330891aa1c95db | 127.29.180.126:41303 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0702aa63d2a34898b13be46470a6dd1f | 127.29.180.65:34975 | HEALTHY | <none> | 0 | 0
aaaed6a8a7ad4c0389a4daad6b2b4dc4 | 127.29.180.66:40163 | HEALTHY | <none> | 4 | 0
c6bf8ae821ac41859a4aebb8ede00632 | 127.29.180.67:33183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:34975
env_inject_eio | 1 | hidden | 127.29.180.65:34975
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:34975
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:34975
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:40163
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:33183
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:34975
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:40163
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:33183
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet e319d6c4dac04f1b9856214abe251b74 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): RUNNING
All reported replicas are:
A = aaaed6a8a7ad4c0389a4daad6b2b4dc4
B = c6bf8ae821ac41859a4aebb8ede00632
C = 0702aa63d2a34898b13be46470a6dd1f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 5 | Yes
B | A* B C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
Tablet 6482d8c7bc374c88826a3e3f295c2021 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
aaaed6a8a7ad4c0389a4daad6b2b4dc4 (127.29.180.66:40163): RUNNING [LEADER]
c6bf8ae821ac41859a4aebb8ede00632 (127.29.180.67:33183): RUNNING
0702aa63d2a34898b13be46470a6dd1f (127.29.180.65:34975): RUNNING
All reported replicas are:
A = aaaed6a8a7ad4c0389a4daad6b2b4dc4
B = c6bf8ae821ac41859a4aebb8ede00632
C = 0702aa63d2a34898b13be46470a6dd1f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 0 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:48:41.918865 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:48:41.987157 619 raft_consensus.cc:1217] T e319d6c4dac04f1b9856214abe251b74 P 0702aa63d2a34898b13be46470a6dd1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260808 01:48:42.430722 598 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:48:42.441349 32666 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:48:42.443423 338 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
89c7916fdcd048d58b330891aa1c95db | 127.29.180.126:41303 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0702aa63d2a34898b13be46470a6dd1f | 127.29.180.65:34975 | HEALTHY | <none> | 0 | 0
aaaed6a8a7ad4c0389a4daad6b2b4dc4 | 127.29.180.66:40163 | HEALTHY | <none> | 4 | 0
c6bf8ae821ac41859a4aebb8ede00632 | 127.29.180.67:33183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:34975
env_inject_eio | 1 | hidden | 127.29.180.65:34975
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:34975
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:34975
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:40163
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:33183
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:34975
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:40163
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:33183
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:42407 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:48:42.716042 30417 log_verifier.cc:126] Checking tablet 6482d8c7bc374c88826a3e3f295c2021
I20260808 01:48:42.782887 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6482d8c7bc374c88826a3e3f295c2021
I20260808 01:48:42.783108 30417 log_verifier.cc:126] Checking tablet b1fcf8b4e5b1403b97bad94682add9e5
I20260808 01:48:42.846210 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b1fcf8b4e5b1403b97bad94682add9e5
I20260808 01:48:42.846441 30417 log_verifier.cc:126] Checking tablet c5181f3f52b649e2847c9b9107d3f7f5
I20260808 01:48:42.906529 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c5181f3f52b649e2847c9b9107d3f7f5
I20260808 01:48:42.906886 30417 log_verifier.cc:126] Checking tablet e319d6c4dac04f1b9856214abe251b74
I20260808 01:48:42.975302 30417 log_verifier.cc:177] Verified matching terms for 5 ops in tablet e319d6c4dac04f1b9856214abe251b74
I20260808 01:48:42.975554 30417 log_verifier.cc:126] Checking tablet ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:43.042261 30417 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ef18b6b84f7b40419d7f568744c17b24
I20260808 01:48:43.458300 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 504
W20260808 01:48:43.500373 32618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108) [suppressed 19 similar messages]
I20260808 01:48:43.501575 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 32595
I20260808 01:48:43.542755 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 32736
I20260808 01:48:43.587149 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 32351
2026-08-08T01:48:43Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2 (16410 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3
2026-08-08T01:48:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:48:43Z Disabled control of system clock
I20260808 01:48:43.721076 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:43261
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:43261
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:48:44.052155 797 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:44.052769 797 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:44.053169 797 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:44.088007 797 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:48:44.088315 797 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:44.088534 797 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:48:44.088737 797 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:48:44.128491 797 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:43261
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:44.129926 797 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:44.131731 797 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:44.143597 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:45.547746 802 debug-util.cc:398] Leaking SignalData structure 0x7b08000290e0 after lost signal to thread 797
W20260808 01:48:45.980672 802 kernel_stack_watchdog.cc:198] Thread 797 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:45.981051 797 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.837s user 0.617s sys 1.220s
W20260808 01:48:45.981539 797 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.838s user 0.617s sys 1.220s
W20260808 01:48:44.144254 804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:45.983598 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:45.986183 805 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1838 milliseconds
I20260808 01:48:45.986208 797 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:45.987584 797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:45.990231 797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:45.991626 797 hybrid_clock.cc:648] HybridClock initialized: now 1786153725991583 us; error 49 us; skew 500 ppm
I20260808 01:48:45.992486 797 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:45.999385 797 webserver.cc:533] Webserver started at http://127.29.180.126:40555/ using document root <none> and password file <none>
I20260808 01:48:46.000964 797 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:46.001235 797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:46.001979 797 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:46.006834 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.008034 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.008962 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.009943 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.010836 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.011727 797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.036625 797 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.013s sys 0.013s
I20260808 01:48:46.045004 818 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:46.045578 818 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:46.046768 818 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:46.046972 819 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:46.047135 818 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:46.049516 797 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.004s sys 0.003s
I20260808 01:48:46.049966 797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "8285c8a80e3e4985a26f56fbac0d4b97"
format_stamp: "Formatted at 2026-08-08 01:48:46 on dist-test-slave-p3t5"
I20260808 01:48:46.050385 797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:46.108371 797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:46.110028 797 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:46.110539 797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:46.184811 797 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:43261
I20260808 01:48:46.184922 872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:43261 every 8 connection(s)
I20260808 01:48:46.187911 797 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:48:46.190709 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 797
I20260808 01:48:46.191378 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:48:46.222183 873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97: Bootstrap starting.
I20260808 01:48:46.228165 873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:46.230182 873 log.cc:826] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:46.235095 873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97: No bootstrap required, opened a new log
I20260808 01:48:46.253154 873 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } }
I20260808 01:48:46.254048 873 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:46.254406 873 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8285c8a80e3e4985a26f56fbac0d4b97, State: Initialized, Role: FOLLOWER
I20260808 01:48:46.255292 873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } }
I20260808 01:48:46.255856 873 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:46.256209 873 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:46.256587 873 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:46.261065 873 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } }
I20260808 01:48:46.261914 873 leader_election.cc:304] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8285c8a80e3e4985a26f56fbac0d4b97; no voters:
I20260808 01:48:46.263965 873 leader_election.cc:290] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:48:46.264474 878 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:46.267262 878 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [term 1 LEADER]: Becoming Leader. State: Replica: 8285c8a80e3e4985a26f56fbac0d4b97, State: Running, Role: LEADER
I20260808 01:48:46.268090 878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } }
I20260808 01:48:46.269089 873 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:46.278208 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8285c8a80e3e4985a26f56fbac0d4b97. Latest consensus state: current_term: 1 leader_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } } }
I20260808 01:48:46.280406 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:46.279510 879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8285c8a80e3e4985a26f56fbac0d4b97" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 43261 } } }
I20260808 01:48:46.281949 879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97 [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:46.284226 888 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:46.298370 888 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:46.316390 888 catalog_manager.cc:1383] Generated new cluster ID: 715eafaeadae42d78647d2394d755cbf
I20260808 01:48:46.316803 888 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:46.348217 888 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:46.350366 888 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:46.365852 888 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8285c8a80e3e4985a26f56fbac0d4b97: Generated new TSK 0
I20260808 01:48:46.367075 888 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:46.391709 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:48:46.730324 898 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:46.730898 898 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:46.731393 898 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:46.766083 898 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:46.767030 898 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:46.806780 898 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:46.808298 898 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:46.810205 898 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:46.824831 904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:48.229032 903 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 898
W20260808 01:48:48.753679 898 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.926s user 0.001s sys 0.000s
W20260808 01:48:46.826107 905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:48.754840 898 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.928s user 0.002s sys 0.000s
W20260808 01:48:48.754776 907 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1927 milliseconds
W20260808 01:48:48.756371 909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:48.756325 898 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:48.760672 898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:48.763445 898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:48.764964 898 hybrid_clock.cc:648] HybridClock initialized: now 1786153728764917 us; error 44 us; skew 500 ppm
I20260808 01:48:48.766151 898 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:48.774286 898 webserver.cc:533] Webserver started at http://127.29.180.65:44481/ using document root <none> and password file <none>
I20260808 01:48:48.776539 898 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:48.776854 898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:48.777796 898 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:48.784413 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.786099 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.787398 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.788609 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.789902 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.791117 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.820887 898 fs_manager.cc:696] Time spent creating directory manager: real 0.029s user 0.021s sys 0.009s
I20260808 01:48:48.829908 919 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.830586 919 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.831518 919 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.831707 920 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.831965 919 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.834406 898 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260808 01:48:48.834775 898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:48.835201 898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:48.900656 898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:48.902311 898 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:48.902771 898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:48.906160 898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:48.911389 898 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:48.911592 898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:48.911936 898 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:48.912115 898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:49.089999 898 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:33489
I20260808 01:48:49.090118 1036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:33489 every 8 connection(s)
I20260808 01:48:49.092887 898 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:48:49.097442 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 898
I20260808 01:48:49.098220 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:48:49.107919 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:49.142316 1037 heartbeater.cc:344] Connected to a master server at 127.29.180.126:43261
I20260808 01:48:49.142828 1037 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:49.143987 1037 heartbeater.cc:507] Master 127.29.180.126:43261 requested a full tablet report, sending...
I20260808 01:48:49.146847 838 ts_manager.cc:194] Registered new tserver with Master: 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489)
I20260808 01:48:49.148996 838 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:39597
W20260808 01:48:49.456226 1041 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:49.456734 1041 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:49.457166 1041 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:49.492626 1041 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:49.493443 1041 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:48:49.530081 1041 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:49.531478 1041 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:49.533133 1041 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:49.546700 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:50.153455 1037 heartbeater.cc:499] Master 127.29.180.126:43261 was elected leader, sending a full tablet report...
W20260808 01:48:50.965286 1046 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 1041
W20260808 01:48:49.547132 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:51.014140 1046 kernel_stack_watchdog.cc:198] Thread 1041 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 415ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:51.014865 1041 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.468s user 0.000s sys 0.002s
W20260808 01:48:51.015158 1041 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.468s user 0.000s sys 0.002s
W20260808 01:48:51.016479 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:51.018800 1049 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1465 milliseconds
I20260808 01:48:51.018883 1041 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:51.020243 1041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:51.022482 1041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:51.023919 1041 hybrid_clock.cc:648] HybridClock initialized: now 1786153731023882 us; error 41 us; skew 500 ppm
I20260808 01:48:51.024843 1041 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:51.032254 1041 webserver.cc:533] Webserver started at http://127.29.180.66:33477/ using document root <none> and password file <none>
I20260808 01:48:51.033937 1041 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:51.034224 1041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:51.034934 1041 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:51.039598 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.040968 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.042008 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.042928 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.043898 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.044795 1041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.068868 1041 fs_manager.cc:696] Time spent creating directory manager: real 0.023s user 0.018s sys 0.008s
I20260808 01:48:51.077315 1061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.078230 1061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.078845 1061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.079053 1062 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.079259 1061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.081503 1041 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.000s sys 0.006s
I20260808 01:48:51.081930 1041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "2b4d34512c9f4851b029ef32c177d152"
format_stamp: "Formatted at 2026-08-08 01:48:51 on dist-test-slave-p3t5"
I20260808 01:48:51.082350 1041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:51.143119 1041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:51.144718 1041 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:51.145203 1041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:51.148084 1041 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:51.152747 1041 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:51.153007 1041 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.153288 1041 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:51.153496 1041 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:51.299013 1041 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:36125
I20260808 01:48:51.299124 1176 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:36125 every 8 connection(s)
I20260808 01:48:51.301851 1041 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:48:51.307390 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 1041
I20260808 01:48:51.308059 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:48:51.316717 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:51.327366 1177 heartbeater.cc:344] Connected to a master server at 127.29.180.126:43261
I20260808 01:48:51.327884 1177 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:51.329023 1177 heartbeater.cc:507] Master 127.29.180.126:43261 requested a full tablet report, sending...
I20260808 01:48:51.331557 838 ts_manager.cc:194] Registered new tserver with Master: 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125)
I20260808 01:48:51.332857 838 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:60635
W20260808 01:48:51.647784 1181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:51.648399 1181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:51.649014 1181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:51.683169 1181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:51.684262 1181 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:48:51.723934 1181 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:51.725808 1181 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:51.728327 1181 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:51.744020 1187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:52.336369 1177 heartbeater.cc:499] Master 127.29.180.126:43261 was elected leader, sending a full tablet report...
W20260808 01:48:53.148598 1186 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 1181
W20260808 01:48:51.744537 1188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:53.466779 1181 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.722s user 0.612s sys 1.104s
W20260808 01:48:53.467182 1181 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.723s user 0.612s sys 1.104s
W20260808 01:48:53.469350 1190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:53.472007 1189 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1721 milliseconds
I20260808 01:48:53.472070 1181 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:53.473492 1181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:53.475679 1181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:53.477036 1181 hybrid_clock.cc:648] HybridClock initialized: now 1786153733477000 us; error 53 us; skew 500 ppm
I20260808 01:48:53.477919 1181 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:53.484963 1181 webserver.cc:533] Webserver started at http://127.29.180.67:33813/ using document root <none> and password file <none>
I20260808 01:48:53.487043 1181 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:53.487325 1181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:53.488103 1181 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:53.492697 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.494035 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.494978 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.495882 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.496762 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.497723 1181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.522445 1181 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.017s sys 0.006s
I20260808 01:48:53.530656 1201 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.531440 1201 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.532081 1201 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.532377 1202 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.532508 1201 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.535267 1181 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.002s
I20260808 01:48:53.535588 1181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "051fc57f839541198f282afd3fcc20db"
format_stamp: "Formatted at 2026-08-08 01:48:53 on dist-test-slave-p3t5"
I20260808 01:48:53.535957 1181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:53.615288 1181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:53.616840 1181 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:53.617280 1181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:53.620309 1181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:53.625348 1181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:53.625571 1181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.625908 1181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:53.626091 1181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:53.772154 1181 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:37635
I20260808 01:48:53.772516 1318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:37635 every 8 connection(s)
I20260808 01:48:53.775465 1181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:48:53.784655 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 1181
I20260808 01:48:53.785148 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:48:53.792501 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.68:0
--local_ip_for_outbound_sockets=127.29.180.68
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:48:53.801028 1319 heartbeater.cc:344] Connected to a master server at 127.29.180.126:43261
I20260808 01:48:53.801601 1319 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:53.802910 1319 heartbeater.cc:507] Master 127.29.180.126:43261 requested a full tablet report, sending...
I20260808 01:48:53.805502 838 ts_manager.cc:194] Registered new tserver with Master: 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635)
I20260808 01:48:53.807511 838 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:39195
W20260808 01:48:54.120944 1323 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:54.121460 1323 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:54.121927 1323 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:54.155283 1323 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:54.156193 1323 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.68
I20260808 01:48:54.197099 1323 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:43261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:54.198589 1323 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:48:54.200326 1323 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:54.213275 1330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:54.811618 1319 heartbeater.cc:499] Master 127.29.180.126:43261 was elected leader, sending a full tablet report...
W20260808 01:48:55.617715 1329 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 1323
W20260808 01:48:55.984812 1323 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.771s user 0.616s sys 1.136s
W20260808 01:48:54.215322 1331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:55.985221 1323 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.771s user 0.616s sys 1.136s
W20260808 01:48:55.987812 1333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:55.993193 1332 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1777 milliseconds
I20260808 01:48:55.993254 1323 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:55.994570 1323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:55.996786 1323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:55.998155 1323 hybrid_clock.cc:648] HybridClock initialized: now 1786153735998133 us; error 62 us; skew 500 ppm
I20260808 01:48:55.999032 1323 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:56.006073 1323 webserver.cc:533] Webserver started at http://127.29.180.68:36507/ using document root <none> and password file <none>
I20260808 01:48:56.007654 1323 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:56.007951 1323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:56.008646 1323 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:56.013236 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.014559 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-1/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.015496 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-2/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.016456 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-3/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.017367 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-4/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.018362 1323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance:
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.042891 1323 fs_manager.cc:696] Time spent creating directory manager: real 0.024s user 0.010s sys 0.014s
I20260808 01:48:56.051702 1344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.052476 1344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.053283 1344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.053283 1345 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.053835 1344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.056541 1323 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.008s sys 0.001s
I20260808 01:48:56.056931 1323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal
uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-p3t5"
I20260808 01:48:56.057363 1323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:56.119421 1323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:56.121165 1323 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:56.121690 1323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:56.124506 1323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:56.129122 1323 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:56.129376 1323 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.129693 1323 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:56.129894 1323 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:56.277354 1323 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.68:37037
I20260808 01:48:56.277472 1461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.68:37037 every 8 connection(s)
I20260808 01:48:56.280097 1323 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
I20260808 01:48:56.290642 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 1323
I20260808 01:48:56.291136 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance
I20260808 01:48:56.309933 1462 heartbeater.cc:344] Connected to a master server at 127.29.180.126:43261
I20260808 01:48:56.310514 1462 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:56.311985 1462 heartbeater.cc:507] Master 127.29.180.126:43261 requested a full tablet report, sending...
I20260808 01:48:56.314813 838 ts_manager.cc:194] Registered new tserver with Master: 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:48:56.316332 838 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.68:51967
I20260808 01:48:56.329398 30417 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20260808 01:48:56.412683 837 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33070:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:48:56.499228 1251 tablet_service.cc:1511] Processing CreateTablet for tablet 4578d3e670ce40bbb917de3de94c80a2 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:56.506340 1395 tablet_service.cc:1511] Processing CreateTablet for tablet 4578d3e670ce40bbb917de3de94c80a2 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:56.510013 1393 tablet_service.cc:1511] Processing CreateTablet for tablet 335fdcca02cd4457b5e220acc4c9bb08 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:56.510784 1250 tablet_service.cc:1511] Processing CreateTablet for tablet b5b3c8b264e34085ac8839bd8668caa8 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:56.515177 1394 tablet_service.cc:1511] Processing CreateTablet for tablet d434eb82ce0e4369bcc673450e99f368 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:56.518011 968 tablet_service.cc:1511] Processing CreateTablet for tablet b5b3c8b264e34085ac8839bd8668caa8 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:56.519047 1109 tablet_service.cc:1511] Processing CreateTablet for tablet 335fdcca02cd4457b5e220acc4c9bb08 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:56.520183 1249 tablet_service.cc:1511] Processing CreateTablet for tablet 335fdcca02cd4457b5e220acc4c9bb08 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:48:56.521584 1248 tablet_service.cc:1511] Processing CreateTablet for tablet 369b6f293ec74f32b02e936e6539461a (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:56.522868 1110 tablet_service.cc:1511] Processing CreateTablet for tablet d434eb82ce0e4369bcc673450e99f368 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:56.522817 1111 tablet_service.cc:1511] Processing CreateTablet for tablet b5b3c8b264e34085ac8839bd8668caa8 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:48:56.525295 969 tablet_service.cc:1511] Processing CreateTablet for tablet 4578d3e670ce40bbb917de3de94c80a2 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:48:56.525118 1108 tablet_service.cc:1511] Processing CreateTablet for tablet 369b6f293ec74f32b02e936e6539461a (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:56.527632 966 tablet_service.cc:1511] Processing CreateTablet for tablet 369b6f293ec74f32b02e936e6539461a (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:48:56.527643 967 tablet_service.cc:1511] Processing CreateTablet for tablet d434eb82ce0e4369bcc673450e99f368 (DEFAULT_TABLE table=test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:48:56.597868 1488 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
I20260808 01:48:56.601138 1489 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap starting.
I20260808 01:48:56.599159 1487 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152: Bootstrap starting.
I20260808 01:48:56.609247 1491 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
I20260808 01:48:56.611203 1492 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
I20260808 01:48:56.615118 1489 tablet_bootstrap.cc:654] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.616348 1487 tablet_bootstrap.cc:654] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.616557 1493 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap starting.
I20260808 01:48:56.620126 1489 log.cc:826] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:56.620635 1494 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap starting.
I20260808 01:48:56.623368 1495 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db: Bootstrap starting.
I20260808 01:48:56.626441 1490 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
I20260808 01:48:56.644071 1488 tablet_bootstrap.cc:654] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.646466 1490 tablet_bootstrap.cc:654] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.646770 1488 log.cc:826] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:56.632709 1496 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Bootstrap starting.
I20260808 01:48:56.634012 1497 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db: Bootstrap starting.
I20260808 01:48:56.643126 1498 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: Bootstrap starting.
I20260808 01:48:56.653506 1501 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap starting.
I20260808 01:48:56.654747 1491 tablet_bootstrap.cc:654] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.655158 1492 tablet_bootstrap.cc:654] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.665161 1499 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap starting.
I20260808 01:48:56.673988 1500 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap starting.
I20260808 01:48:56.679193 1493 tablet_bootstrap.cc:654] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.682266 1490 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: No bootstrap required, opened a new log
I20260808 01:48:56.682977 1490 ts_tablet_manager.cc:1403] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.011s
I20260808 01:48:56.688949 1495 tablet_bootstrap.cc:654] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.690780 1496 tablet_bootstrap.cc:654] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.693679 1498 tablet_bootstrap.cc:654] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.694226 1488 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: No bootstrap required, opened a new log
I20260808 01:48:56.694766 1488 ts_tablet_manager.cc:1403] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.006s
I20260808 01:48:56.697582 1494 tablet_bootstrap.cc:654] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.699239 1497 tablet_bootstrap.cc:654] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.702788 1496 log.cc:826] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:56.711017 1489 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: No bootstrap required, opened a new log
I20260808 01:48:56.711732 1489 ts_tablet_manager.cc:1403] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: Time spent bootstrapping tablet: real 0.111s user 0.020s sys 0.005s
I20260808 01:48:56.715444 1501 tablet_bootstrap.cc:654] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.719578 1501 log.cc:826] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:56.722520 1496 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: No bootstrap required, opened a new log
I20260808 01:48:56.723120 1496 ts_tablet_manager.cc:1403] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.005s
I20260808 01:48:56.723304 1487 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152: No bootstrap required, opened a new log
I20260808 01:48:56.723883 1487 ts_tablet_manager.cc:1403] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.016s
I20260808 01:48:56.729981 1499 tablet_bootstrap.cc:654] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.731509 1494 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: No bootstrap required, opened a new log
I20260808 01:48:56.732084 1494 ts_tablet_manager.cc:1403] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.011s
I20260808 01:48:56.732506 1491 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: No bootstrap required, opened a new log
I20260808 01:48:56.733142 1491 ts_tablet_manager.cc:1403] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.125s user 0.007s sys 0.008s
I20260808 01:48:56.736333 1492 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: No bootstrap required, opened a new log
I20260808 01:48:56.736852 1492 ts_tablet_manager.cc:1403] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.126s user 0.013s sys 0.006s
I20260808 01:48:56.740239 1500 tablet_bootstrap.cc:654] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.747651 1498 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: No bootstrap required, opened a new log
I20260808 01:48:56.748113 1498 ts_tablet_manager.cc:1403] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: Time spent bootstrapping tablet: real 0.107s user 0.015s sys 0.004s
I20260808 01:48:56.751261 1495 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db: No bootstrap required, opened a new log
I20260808 01:48:56.751724 1495 ts_tablet_manager.cc:1403] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.010s
I20260808 01:48:56.760190 1493 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: No bootstrap required, opened a new log
I20260808 01:48:56.760664 1493 ts_tablet_manager.cc:1403] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: Time spent bootstrapping tablet: real 0.145s user 0.032s sys 0.013s
I20260808 01:48:56.762193 1499 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: No bootstrap required, opened a new log
I20260808 01:48:56.762764 1499 ts_tablet_manager.cc:1403] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.011s
I20260808 01:48:56.765588 1497 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db: No bootstrap required, opened a new log
I20260808 01:48:56.766045 1497 ts_tablet_manager.cc:1403] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db: Time spent bootstrapping tablet: real 0.132s user 0.016s sys 0.005s
I20260808 01:48:56.796001 1487 raft_consensus.cc:359] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.797086 1487 raft_consensus.cc:385] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.797485 1487 raft_consensus.cc:740] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.798593 1487 consensus_queue.cc:260] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.800493 1501 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: No bootstrap required, opened a new log
I20260808 01:48:56.801867 1501 ts_tablet_manager.cc:1403] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent bootstrapping tablet: real 0.149s user 0.035s sys 0.047s
I20260808 01:48:56.802341 1493 raft_consensus.cc:359] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.803236 1493 raft_consensus.cc:385] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.803579 1493 raft_consensus.cc:740] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.804425 1493 consensus_queue.cc:260] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.804703 1495 raft_consensus.cc:359] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.805828 1495 raft_consensus.cc:385] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.806160 1495 raft_consensus.cc:740] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.806450 1496 raft_consensus.cc:359] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.808094 1496 raft_consensus.cc:385] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.808442 1496 raft_consensus.cc:740] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.808292 1495 consensus_queue.cc:260] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.809448 1496 consensus_queue.cc:260] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.807309 1497 raft_consensus.cc:359] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.807471 1494 raft_consensus.cc:359] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.810658 1497 raft_consensus.cc:385] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.810969 1494 raft_consensus.cc:385] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.810979 1497 raft_consensus.cc:740] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.811311 1494 raft_consensus.cc:740] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.811985 1497 consensus_queue.cc:260] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.814755 1494 consensus_queue.cc:260] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.820953 1500 tablet_bootstrap.cc:492] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: No bootstrap required, opened a new log
I20260808 01:48:56.821453 1500 ts_tablet_manager.cc:1403] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent bootstrapping tablet: real 0.148s user 0.012s sys 0.004s
I20260808 01:48:56.825536 1492 raft_consensus.cc:359] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.827032 1492 raft_consensus.cc:385] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.827456 1492 raft_consensus.cc:740] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.828971 1492 consensus_queue.cc:260] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.831838 1501 raft_consensus.cc:359] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.832418 1490 raft_consensus.cc:359] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.833117 1490 raft_consensus.cc:385] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.833350 1501 raft_consensus.cc:385] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.833904 1501 raft_consensus.cc:740] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a462e1c7eb749fd8f898f9a85c8b691, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.834447 1491 raft_consensus.cc:359] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.834553 1499 raft_consensus.cc:359] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.835143 1491 raft_consensus.cc:385] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.835441 1491 raft_consensus.cc:740] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.835918 1501 consensus_queue.cc:260] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.836306 1491 consensus_queue.cc:260] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.837226 1494 ts_tablet_manager.cc:1434] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: Time spent starting tablet: real 0.105s user 0.012s sys 0.005s
I20260808 01:48:56.833420 1490 raft_consensus.cc:740] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.838435 1490 consensus_queue.cc:260] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.835264 1499 raft_consensus.cc:385] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.840633 1499 raft_consensus.cc:740] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a462e1c7eb749fd8f898f9a85c8b691, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.840734 1488 raft_consensus.cc:359] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.841442 1488 raft_consensus.cc:385] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.841759 1488 raft_consensus.cc:740] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.844676 1493 ts_tablet_manager.cc:1434] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: Time spent starting tablet: real 0.084s user 0.005s sys 0.009s
I20260808 01:48:56.846096 1489 raft_consensus.cc:359] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.847046 1489 raft_consensus.cc:385] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.847388 1489 raft_consensus.cc:740] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.842545 1499 consensus_queue.cc:260] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.851527 1500 raft_consensus.cc:359] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.852304 1500 raft_consensus.cc:385] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.852589 1500 raft_consensus.cc:740] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a462e1c7eb749fd8f898f9a85c8b691, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.854687 1487 ts_tablet_manager.cc:1434] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152: Time spent starting tablet: real 0.130s user 0.009s sys 0.006s
I20260808 01:48:56.857306 1462 heartbeater.cc:499] Master 127.29.180.126:43261 was elected leader, sending a full tablet report...
I20260808 01:48:56.860505 1501 ts_tablet_manager.cc:1434] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent starting tablet: real 0.058s user 0.013s sys 0.002s
I20260808 01:48:56.863185 1499 ts_tablet_manager.cc:1434] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent starting tablet: real 0.100s user 0.035s sys 0.005s
I20260808 01:48:56.863406 1491 ts_tablet_manager.cc:1434] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Time spent starting tablet: real 0.130s user 0.013s sys 0.001s
I20260808 01:48:56.845499 1488 consensus_queue.cc:260] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.864662 1489 consensus_queue.cc:260] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:56.866609 1488 ts_tablet_manager.cc:1434] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Time spent starting tablet: real 0.171s user 0.012s sys 0.006s
I20260808 01:48:56.866900 1489 ts_tablet_manager.cc:1434] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152: Time spent starting tablet: real 0.155s user 0.028s sys 0.007s
I20260808 01:48:56.867549 1500 consensus_queue.cc:260] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.868955 1492 ts_tablet_manager.cc:1434] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Time spent starting tablet: real 0.132s user 0.009s sys 0.003s
I20260808 01:48:56.870223 1500 ts_tablet_manager.cc:1434] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent starting tablet: real 0.048s user 0.004s sys 0.003s
W20260808 01:48:56.880864 1038 tablet.cc:2406] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:56.861654 1490 ts_tablet_manager.cc:1434] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Time spent starting tablet: real 0.178s user 0.015s sys 0.009s
I20260808 01:48:56.887354 1495 ts_tablet_manager.cc:1434] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db: Time spent starting tablet: real 0.135s user 0.009s sys 0.003s
I20260808 01:48:56.893078 1498 raft_consensus.cc:359] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.894013 1498 raft_consensus.cc:385] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.894526 1498 raft_consensus.cc:740] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.895341 1498 consensus_queue.cc:260] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:56.902539 1497 ts_tablet_manager.cc:1434] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db: Time spent starting tablet: real 0.136s user 0.019s sys 0.033s
I20260808 01:48:56.905231 1498 ts_tablet_manager.cc:1434] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: Time spent starting tablet: real 0.157s user 0.017s sys 0.003s
I20260808 01:48:56.905382 1496 ts_tablet_manager.cc:1434] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Time spent starting tablet: real 0.182s user 0.019s sys 0.022s
I20260808 01:48:56.916698 1521 raft_consensus.cc:493] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:56.917326 1521 raft_consensus.cc:515] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.920217 1521 leader_election.cc:290] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125)
I20260808 01:48:56.936582 1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "369b6f293ec74f32b02e936e6539461a" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152" is_pre_election: true
I20260808 01:48:56.936506 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "369b6f293ec74f32b02e936e6539461a" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf" is_pre_election: true
I20260808 01:48:56.937451 1132 raft_consensus.cc:2468] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 0.
I20260808 01:48:56.937513 990 raft_consensus.cc:2468] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 0.
I20260808 01:48:56.939081 1204 leader_election.cc:304] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 2b4d34512c9f4851b029ef32c177d152; no voters:
I20260808 01:48:56.939980 1521 raft_consensus.cc:2804] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:56.940311 1521 raft_consensus.cc:493] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:56.940610 1521 raft_consensus.cc:3060] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.945384 1521 raft_consensus.cc:515] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.947187 1521 leader_election.cc:290] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 election: Requested vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125)
I20260808 01:48:56.948228 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "369b6f293ec74f32b02e936e6539461a" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf"
I20260808 01:48:56.948566 1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "369b6f293ec74f32b02e936e6539461a" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152"
I20260808 01:48:56.948756 990 raft_consensus.cc:3060] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.949158 1132 raft_consensus.cc:3060] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.953801 990 raft_consensus.cc:2468] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 1.
I20260808 01:48:56.953821 1132 raft_consensus.cc:2468] T 369b6f293ec74f32b02e936e6539461a P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 1.
I20260808 01:48:56.954939 1204 leader_election.cc:304] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:56.955652 1521 raft_consensus.cc:2804] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:56.956123 1521 raft_consensus.cc:697] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [term 1 LEADER]: Becoming Leader. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Running, Role: LEADER
I20260808 01:48:56.957091 1521 consensus_queue.cc:237] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.968138 836 catalog_manager.cc:5697] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db reported cstate change: term changed from 0 to 1, leader changed from <none> to 051fc57f839541198f282afd3fcc20db (127.29.180.67). New cstate: current_term: 1 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:56.970793 1518 raft_consensus.cc:493] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:56.971397 1518 raft_consensus.cc:515] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.974843 1518 leader_election.cc:290] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635)
I20260808 01:48:56.987478 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5b3c8b264e34085ac8839bd8668caa8" candidate_uuid: "2b4d34512c9f4851b029ef32c177d152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf" is_pre_election: true
I20260808 01:48:56.988061 990 raft_consensus.cc:2468] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b4d34512c9f4851b029ef32c177d152 in term 0.
I20260808 01:48:56.989271 1064 leader_election.cc:304] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b4d34512c9f4851b029ef32c177d152, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:56.989250 1272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5b3c8b264e34085ac8839bd8668caa8" candidate_uuid: "2b4d34512c9f4851b029ef32c177d152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051fc57f839541198f282afd3fcc20db" is_pre_election: true
I20260808 01:48:56.990221 1518 raft_consensus.cc:2804] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:56.990126 1272 raft_consensus.cc:2468] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b4d34512c9f4851b029ef32c177d152 in term 0.
I20260808 01:48:56.990566 1518 raft_consensus.cc:493] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:56.990908 1518 raft_consensus.cc:3060] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.995874 1518 raft_consensus.cc:515] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:56.997443 1518 leader_election.cc:290] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [CANDIDATE]: Term 1 election: Requested vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635)
I20260808 01:48:56.998270 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5b3c8b264e34085ac8839bd8668caa8" candidate_uuid: "2b4d34512c9f4851b029ef32c177d152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf"
I20260808 01:48:56.998587 1272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5b3c8b264e34085ac8839bd8668caa8" candidate_uuid: "2b4d34512c9f4851b029ef32c177d152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051fc57f839541198f282afd3fcc20db"
I20260808 01:48:56.998733 990 raft_consensus.cc:3060] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.999075 1272 raft_consensus.cc:3060] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.002876 990 raft_consensus.cc:2468] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b4d34512c9f4851b029ef32c177d152 in term 1.
I20260808 01:48:57.003921 1064 leader_election.cc:304] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b4d34512c9f4851b029ef32c177d152, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:57.004253 1272 raft_consensus.cc:2468] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b4d34512c9f4851b029ef32c177d152 in term 1.
I20260808 01:48:57.004609 1518 raft_consensus.cc:2804] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:57.005149 1518 raft_consensus.cc:697] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Running, Role: LEADER
I20260808 01:48:57.006129 1518 consensus_queue.cc:237] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:48:57.018507 1524 raft_consensus.cc:493] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:57.018621 836 catalog_manager.cc:5697] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "2b4d34512c9f4851b029ef32c177d152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:57.018991 1524 raft_consensus.cc:515] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:57.027691 1524 leader_election.cc:290] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
W20260808 01:48:57.034816 1320 tablet.cc:2406] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:57.035109 1272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "7b828feeb83e41289cb6db076965c7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051fc57f839541198f282afd3fcc20db" is_pre_election: true
I20260808 01:48:57.035689 1272 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b828feeb83e41289cb6db076965c7cf in term 0.
W20260808 01:48:57.036345 1463 tablet.cc:2406] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:57.037020 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "7b828feeb83e41289cb6db076965c7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" is_pre_election: true
I20260808 01:48:57.037213 922 leader_election.cc:304] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:57.038064 1524 raft_consensus.cc:2804] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:57.037910 1416 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b828feeb83e41289cb6db076965c7cf in term 0.
I20260808 01:48:57.038376 1524 raft_consensus.cc:493] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:57.038640 1524 raft_consensus.cc:3060] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.043571 1524 raft_consensus.cc:515] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:57.045230 1524 leader_election.cc:290] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [CANDIDATE]: Term 1 election: Requested vote from peers 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:48:57.046021 1272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "7b828feeb83e41289cb6db076965c7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051fc57f839541198f282afd3fcc20db"
I20260808 01:48:57.046612 1272 raft_consensus.cc:3060] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.046504 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "7b828feeb83e41289cb6db076965c7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
I20260808 01:48:57.047125 1416 raft_consensus.cc:3060] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.053696 1272 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b828feeb83e41289cb6db076965c7cf in term 1.
I20260808 01:48:57.054179 1416 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b828feeb83e41289cb6db076965c7cf in term 1.
I20260808 01:48:57.054925 922 leader_election.cc:304] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:57.055648 1524 raft_consensus.cc:2804] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:57.056227 1524 raft_consensus.cc:697] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 1 LEADER]: Becoming Leader. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Running, Role: LEADER
I20260808 01:48:57.057066 1524 consensus_queue.cc:237] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
W20260808 01:48:57.067026 1178 tablet.cc:2406] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:57.067730 836 catalog_manager.cc:5697] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b828feeb83e41289cb6db076965c7cf (127.29.180.65). New cstate: current_term: 1 leader_uuid: "7b828feeb83e41289cb6db076965c7cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:57.104696 1529 raft_consensus.cc:493] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:57.105273 1529 raft_consensus.cc:515] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:57.108564 1529 leader_election.cc:290] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125), 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489)
I20260808 01:48:57.121701 1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d434eb82ce0e4369bcc673450e99f368" candidate_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152" is_pre_election: true
I20260808 01:48:57.122323 1132 raft_consensus.cc:2468] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a462e1c7eb749fd8f898f9a85c8b691 in term 0.
I20260808 01:48:57.123063 1521 raft_consensus.cc:493] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:57.123576 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d434eb82ce0e4369bcc673450e99f368" candidate_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf" is_pre_election: true
I20260808 01:48:57.123561 1521 raft_consensus.cc:515] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:57.123809 1347 leader_election.cc:304] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a462e1c7eb749fd8f898f9a85c8b691, 2b4d34512c9f4851b029ef32c177d152; no voters:
I20260808 01:48:57.124186 990 raft_consensus.cc:2468] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a462e1c7eb749fd8f898f9a85c8b691 in term 0.
I20260808 01:48:57.124732 1529 raft_consensus.cc:2804] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:57.125068 1529 raft_consensus.cc:493] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:57.125394 1529 raft_consensus.cc:3060] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.125948 1521 leader_election.cc:290] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:48:57.128537 1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335fdcca02cd4457b5e220acc4c9bb08" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152" is_pre_election: true
I20260808 01:48:57.129122 1132 raft_consensus.cc:2468] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 0.
I20260808 01:48:57.130420 1204 leader_election.cc:304] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 2b4d34512c9f4851b029ef32c177d152; no voters:
I20260808 01:48:57.131266 1521 raft_consensus.cc:2804] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:57.131606 1521 raft_consensus.cc:493] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:57.131935 1521 raft_consensus.cc:3060] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.133474 1529 raft_consensus.cc:515] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:57.135488 1529 leader_election.cc:290] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [CANDIDATE]: Term 1 election: Requested vote from peers 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125), 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489)
I20260808 01:48:57.137111 1132 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d434eb82ce0e4369bcc673450e99f368" candidate_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152"
I20260808 01:48:57.137764 1132 raft_consensus.cc:3060] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.138219 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335fdcca02cd4457b5e220acc4c9bb08" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" is_pre_election: true
I20260808 01:48:57.138803 1416 raft_consensus.cc:2468] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 0.
I20260808 01:48:57.138630 1521 raft_consensus.cc:515] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:57.141676 1521 leader_election.cc:290] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 election: Requested vote from peers 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:48:57.141525 1131 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335fdcca02cd4457b5e220acc4c9bb08" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b4d34512c9f4851b029ef32c177d152"
I20260808 01:48:57.142243 1131 raft_consensus.cc:3060] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.142594 990 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d434eb82ce0e4369bcc673450e99f368" candidate_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b828feeb83e41289cb6db076965c7cf"
I20260808 01:48:57.142930 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335fdcca02cd4457b5e220acc4c9bb08" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
I20260808 01:48:57.143231 990 raft_consensus.cc:3060] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.143478 1416 raft_consensus.cc:3060] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:57.149511 1131 raft_consensus.cc:2468] T 335fdcca02cd4457b5e220acc4c9bb08 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 1.
I20260808 01:48:57.149976 1416 raft_consensus.cc:2468] T 335fdcca02cd4457b5e220acc4c9bb08 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 1.
I20260808 01:48:57.151000 1204 leader_election.cc:304] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 2b4d34512c9f4851b029ef32c177d152; no voters:
I20260808 01:48:57.152088 990 raft_consensus.cc:2468] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a462e1c7eb749fd8f898f9a85c8b691 in term 1.
I20260808 01:48:57.152827 1521 raft_consensus.cc:2804] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:57.153584 1521 raft_consensus.cc:697] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [term 1 LEADER]: Becoming Leader. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Running, Role: LEADER
I20260808 01:48:57.153273 1347 leader_election.cc:304] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a462e1c7eb749fd8f898f9a85c8b691, 7b828feeb83e41289cb6db076965c7cf; no voters:
I20260808 01:48:57.154444 1132 raft_consensus.cc:2468] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a462e1c7eb749fd8f898f9a85c8b691 in term 1.
I20260808 01:48:57.154800 1529 raft_consensus.cc:2804] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:57.155316 1529 raft_consensus.cc:697] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEADER]: Becoming Leader. State: Replica: 0a462e1c7eb749fd8f898f9a85c8b691, State: Running, Role: LEADER
I20260808 01:48:57.154598 1521 consensus_queue.cc:237] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:57.156243 1529 consensus_queue.cc:237] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } }
I20260808 01:48:57.165583 836 catalog_manager.cc:5697] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db reported cstate change: term changed from 0 to 1, leader changed from <none> to 051fc57f839541198f282afd3fcc20db (127.29.180.67). New cstate: current_term: 1 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:57.171255 838 catalog_manager.cc:5697] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68). New cstate: current_term: 1 leader_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:57.340426 1521 consensus_queue.cc:1048] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:57.356750 1521 consensus_queue.cc:1048] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:57.374133 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 898
I20260808 01:48:57.398303 1518 consensus_queue.cc:1048] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:48:57.416877 1064 connection.cc:573] client connection to 127.29.180.65:33489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 01:48:57.417362 1064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 01:48:57.422264 1537 negotiation.cc:336] Failed RPC negotiation. Trace:
0808 01:48:57.420267 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.29.180.65:33489 (local address 127.29.180.66:39651)
0808 01:48:57.421046 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0808 01:48:57.421262 (+ 216us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 01:48:57.422181 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:33489
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:43261
--webserver_port=44481
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:46031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:48:57.422987 1064 consensus_peers.cc:597] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:57.564170 1347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111)
W20260808 01:48:57.567616 1347 consensus_peers.cc:597] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:57.659025 1529 consensus_queue.cc:1048] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:57.675772 1521 consensus_queue.cc:1048] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:57.692399 1520 consensus_queue.cc:1048] T 335fdcca02cd4457b5e220acc4c9bb08 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:57.773176 1560 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:48:57.774155 1560 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:48:57.774842 1560 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:48:57.811645 1560 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:48:57.812700 1560 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:48:57.855736 1560 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:46031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:33489
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=44481
--tserver_master_addrs=127.29.180.126:43261
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:48:57.857215 1560 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:48:57.858214 1560 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:48:57.858387 1560 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:48:57.859563 1560 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:57.872954 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:57.935895 1204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111)
W20260808 01:48:57.948031 1204 consensus_peers.cc:597] T 369b6f293ec74f32b02e936e6539461a P 051fc57f839541198f282afd3fcc20db -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:58.748903 1522 raft_consensus.cc:493] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.749439 1522 raft_consensus.cc:515] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:58.762297 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" is_pre_election: true
I20260808 01:48:58.763239 1416 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 1.
I20260808 01:48:58.765497 1206 leader_election.cc:304] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 0a462e1c7eb749fd8f898f9a85c8b691; no voters:
I20260808 01:48:58.766887 1521 raft_consensus.cc:2804] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:48:58.767364 1521 raft_consensus.cc:493] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:58.767808 1521 raft_consensus.cc:3060] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:58.785737 1522 leader_election.cc:290] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
W20260808 01:48:58.771148 1204 leader_election.cc:336] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111)
I20260808 01:48:58.785360 1521 raft_consensus.cc:515] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
W20260808 01:48:58.812963 1204 leader_election.cc:336] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111)
I20260808 01:48:58.820845 1416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4578d3e670ce40bbb917de3de94c80a2" candidate_uuid: "051fc57f839541198f282afd3fcc20db" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a462e1c7eb749fd8f898f9a85c8b691"
I20260808 01:48:58.821561 1416 raft_consensus.cc:3060] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:58.831269 1416 raft_consensus.cc:2468] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 051fc57f839541198f282afd3fcc20db in term 2.
I20260808 01:48:58.831895 1521 leader_election.cc:290] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 election: Requested vote from peers 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489), 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:48:58.836776 1206 leader_election.cc:304] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 051fc57f839541198f282afd3fcc20db, 0a462e1c7eb749fd8f898f9a85c8b691; no voters: 7b828feeb83e41289cb6db076965c7cf
I20260808 01:48:58.838279 1521 raft_consensus.cc:2804] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:58.839987 1521 raft_consensus.cc:697] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 LEADER]: Becoming Leader. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Running, Role: LEADER
I20260808 01:48:58.841986 1521 consensus_queue.cc:237] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:48:58.917237 836 catalog_manager.cc:5697] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db reported cstate change: term changed from 1 to 2, leader changed from 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) to 051fc57f839541198f282afd3fcc20db (127.29.180.67). New cstate: current_term: 2 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:59.255443 1416 raft_consensus.cc:1275] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Refusing update from remote peer 051fc57f839541198f282afd3fcc20db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:48:59.258066 1521 consensus_queue.cc:1048] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:48:59.320989 1204 consensus_peers.cc:597] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:59.355088 1596 log.cc:927] Time spent T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20260808 01:48:59.284782 1571 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 1560
W20260808 01:48:59.717160 1571 kernel_stack_watchdog.cc:198] Thread 1560 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 405ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:48:57.874507 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:59.718298 1560 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.845s user 0.592s sys 1.156s
W20260808 01:48:59.718829 1560 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.846s user 0.592s sys 1.157s
W20260808 01:48:59.721170 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:59.723453 1574 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1848 milliseconds
I20260808 01:48:59.723488 1560 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:48:59.724802 1560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:48:59.727046 1560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:48:59.728477 1560 hybrid_clock.cc:648] HybridClock initialized: now 1786153739728438 us; error 39 us; skew 500 ppm
I20260808 01:48:59.729413 1560 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:59.736119 1560 webserver.cc:533] Webserver started at http://127.29.180.65:44481/ using document root <none> and password file <none>
I20260808 01:48:59.738003 1560 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:59.738304 1560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:48:59.740862 1560 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:48:59.745931 1560 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:48:59.761401 1560 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:48:59.773780 1560 fs_manager.cc:714] Time spent opening directory manager: real 0.030s user 0.027s sys 0.004s
E20260808 01:48:59.781273 1560 log_block_manager.cc:3241] Not using report from /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260808 01:48:59.781667 1610 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:59.782514 1610 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:59.783658 1610 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:59.783763 1611 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:48:59.786851 1560 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.008s sys 0.002s
I20260808 01:48:59.787420 1560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "7b828feeb83e41289cb6db076965c7cf"
format_stamp: "Formatted at 2026-08-08 01:48:48 on dist-test-slave-p3t5"
I20260808 01:48:59.788359 1560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:59.852708 1560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:48:59.854489 1560 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:48:59.855037 1560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:59.857971 1560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:59.864439 1618 ts_tablet_manager.cc:548] Loading tablet metadata (0/4 complete)
W20260808 01:48:59.880475 1620 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 4578d3e670ce40bbb917de3de94c80a2: could not find data dir with uuid 88cb30bcec4545eaa8f65c4435e6bd8d
W20260808 01:48:59.884424 1619 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet d434eb82ce0e4369bcc673450e99f368: could not find data dir with uuid 88cb30bcec4545eaa8f65c4435e6bd8d
W20260808 01:48:59.885535 1347 consensus_peers.cc:597] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:48:59.887950 1621 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet b5b3c8b264e34085ac8839bd8668caa8: could not find data dir with uuid 88cb30bcec4545eaa8f65c4435e6bd8d
I20260808 01:48:59.888614 1560 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260808 01:48:59.888875 1560 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.026s user 0.008s sys 0.008s
I20260808 01:48:59.889178 1560 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
W20260808 01:48:59.891963 869 debug-util.cc:398] Leaking SignalData structure 0x7b08000aeb20 after lost signal to thread 798
W20260808 01:48:59.892730 869 debug-util.cc:398] Leaking SignalData structure 0x7b0800083340 after lost signal to thread 872
I20260808 01:48:59.895664 1621 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
I20260808 01:48:59.899439 1619 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
E20260808 01:48:59.900292 1619 ts_tablet_manager.cc:1425] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet d434eb82ce0e4369bcc673450e99f368
I20260808 01:48:59.900720 1619 ts_tablet_manager.cc:1403] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 01:48:59.901106 1619 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:59.901517 1619 raft_consensus.cc:2243] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:59.904824 1619 raft_consensus.cc:2272] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:59.905469 1620 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
E20260808 01:48:59.905929 1620 ts_tablet_manager.cc:1425] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 4578d3e670ce40bbb917de3de94c80a2
I20260808 01:48:59.906224 1620 ts_tablet_manager.cc:1403] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 01:48:59.906491 1620 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:59.906726 1620 raft_consensus.cc:2243] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:59.907119 1620 raft_consensus.cc:2272] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:59.913101 1560 ts_tablet_manager.cc:616] Registered 4 tablets
I20260808 01:48:59.913445 1560 ts_tablet_manager.cc:595] Time spent register tablets: real 0.024s user 0.022s sys 0.002s
I20260808 01:48:59.913445 1620 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Bootstrap starting.
E20260808 01:48:59.914068 1620 ts_tablet_manager.cc:1425] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:48:59.914400 1620 ts_tablet_manager.cc:1403] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 01:48:59.914701 1620 tablet_replica.cc:333] stopping tablet replica
I20260808 01:48:59.914988 1620 raft_consensus.cc:2243] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:59.915391 1620 raft_consensus.cc:2272] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:59.972287 1621 log.cc:826] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Log is configured to *not* fsync() on all Append() calls
I20260808 01:48:59.985204 1621 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:48:59.987025 1621 tablet_bootstrap.cc:492] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Bootstrap complete.
I20260808 01:48:59.989473 1621 ts_tablet_manager.cc:1403] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Time spent bootstrapping tablet: real 0.094s user 0.054s sys 0.029s
W20260808 01:49:00.019662 1064 consensus_peers.cc:597] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 -> Peer 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Couldn't send request to peer 7b828feeb83e41289cb6db076965c7cf. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:49:00.034066 1621 raft_consensus.cc:359] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:49:00.035449 1621 raft_consensus.cc:740] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b828feeb83e41289cb6db076965c7cf, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.036975 1621 consensus_queue.cc:260] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b828feeb83e41289cb6db076965c7cf" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33489 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:49:00.043751 1621 ts_tablet_manager.cc:1434] T 369b6f293ec74f32b02e936e6539461a P 7b828feeb83e41289cb6db076965c7cf: Time spent starting tablet: real 0.054s user 0.040s sys 0.007s
I20260808 01:49:00.155385 1560 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:33489
I20260808 01:49:00.155712 1735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:33489 every 8 connection(s)
I20260808 01:49:00.158396 1560 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:49:00.165989 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 1560
I20260808 01:49:00.197875 1736 heartbeater.cc:344] Connected to a master server at 127.29.180.126:43261
I20260808 01:49:00.198388 1736 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:00.199541 1736 heartbeater.cc:507] Master 127.29.180.126:43261 requested a full tablet report, sending...
I20260808 01:49:00.204322 836 ts_manager.cc:194] Re-registered known tserver with Master: 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489)
W20260808 01:49:00.206527 836 catalog_manager.cc:5610] Tablet d434eb82ce0e4369bcc673450e99f368 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet d434eb82ce0e4369bcc673450e99f368
W20260808 01:49:00.206995 836 catalog_manager.cc:5610] Tablet 4578d3e670ce40bbb917de3de94c80a2 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 4578d3e670ce40bbb917de3de94c80a2
W20260808 01:49:00.207367 836 catalog_manager.cc:5610] Tablet b5b3c8b264e34085ac8839bd8668caa8 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:00.209780 836 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:47773
I20260808 01:49:00.391601 1347 consensus_queue.cc:579] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: The tablet replica hosted on peer 7b828feeb83e41289cb6db076965c7cf has failed
I20260808 01:49:00.403815 1415 consensus_queue.cc:237] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
I20260808 01:49:00.413195 1131 raft_consensus.cc:1275] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Refusing update from remote peer 0a462e1c7eb749fd8f898f9a85c8b691: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:00.415179 1563 consensus_queue.cc:1048] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:00.425231 1563 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } }
I20260808 01:49:00.428225 1131 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } }
I20260808 01:49:00.437430 823 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d434eb82ce0e4369bcc673450e99f368 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:00.443111 836 catalog_manager.cc:5697] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 reported cstate change: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:00.471099 1415 consensus_queue.cc:237] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } }
I20260808 01:49:00.484895 1670 tablet_service.cc:1558] Processing DeleteTablet for tablet d434eb82ce0e4369bcc673450e99f368 with delete_type TABLET_DATA_TOMBSTONED (TS 7b828feeb83e41289cb6db076965c7cf not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:51382
I20260808 01:49:00.492434 1131 raft_consensus.cc:1275] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Refusing update from remote peer 0a462e1c7eb749fd8f898f9a85c8b691: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:00.496179 1749 consensus_queue.cc:1048] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:00.503621 1757 ts_tablet_manager.cc:1916] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:00.504798 1749 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 051fc57f839541198f282afd3fcc20db (127.29.180.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } } }
I20260808 01:49:00.511489 1131 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 051fc57f839541198f282afd3fcc20db (127.29.180.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } } }
W20260808 01:49:00.514222 1347 consensus_peers.cc:597] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 -> Peer 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635): Couldn't send request to peer 051fc57f839541198f282afd3fcc20db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d434eb82ce0e4369bcc673450e99f368. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:00.526634 1757 ts_tablet_manager.cc:1929] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:00.527305 1757 log.cc:1199] T d434eb82ce0e4369bcc673450e99f368 P 7b828feeb83e41289cb6db076965c7cf: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/d434eb82ce0e4369bcc673450e99f368
I20260808 01:49:00.529881 821 catalog_manager.cc:5028] TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): tablet d434eb82ce0e4369bcc673450e99f368 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) successfully deleted
I20260808 01:49:00.533973 823 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d434eb82ce0e4369bcc673450e99f368 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:49:00.533931 836 catalog_manager.cc:5697] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 reported cstate change: config changed from index 2 to 3, NON_VOTER 051fc57f839541198f282afd3fcc20db (127.29.180.67) added. New cstate: current_term: 1 leader_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } } }
I20260808 01:49:00.562515 1204 consensus_queue.cc:579] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: The tablet replica hosted on peer 7b828feeb83e41289cb6db076965c7cf has failed
I20260808 01:49:00.570402 1736 heartbeater.cc:499] Master 127.29.180.126:43261 was elected leader, sending a full tablet report...
I20260808 01:49:00.577589 1272 consensus_queue.cc:237] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } }
W20260808 01:49:00.580127 838 catalog_manager.cc:5610] Tablet d434eb82ce0e4369bcc673450e99f368 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet d434eb82ce0e4369bcc673450e99f368
W20260808 01:49:00.580821 838 catalog_manager.cc:5610] Tablet 4578d3e670ce40bbb917de3de94c80a2 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 4578d3e670ce40bbb917de3de94c80a2
W20260808 01:49:00.581377 838 catalog_manager.cc:5610] Tablet b5b3c8b264e34085ac8839bd8668caa8 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) has failed on TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:00.590515 1415 raft_consensus.cc:1275] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Refusing update from remote peer 051fc57f839541198f282afd3fcc20db: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:00.593432 1522 consensus_queue.cc:1048] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:00.602250 1521 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } }
I20260808 01:49:00.612300 1416 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } }
I20260808 01:49:00.634609 838 catalog_manager.cc:5697] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 reported cstate change: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } }
I20260808 01:49:00.644395 821 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4578d3e670ce40bbb917de3de94c80a2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:00.648954 1670 tablet_service.cc:1558] Processing DeleteTablet for tablet 4578d3e670ce40bbb917de3de94c80a2 with delete_type TABLET_DATA_TOMBSTONED (TS 7b828feeb83e41289cb6db076965c7cf not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:51382
I20260808 01:49:00.650583 1757 ts_tablet_manager.cc:1916] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:00.673074 1272 consensus_queue.cc:237] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } }
I20260808 01:49:00.681565 1757 ts_tablet_manager.cc:1929] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:00.682066 1757 log.cc:1199] T 4578d3e670ce40bbb917de3de94c80a2 P 7b828feeb83e41289cb6db076965c7cf: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/4578d3e670ce40bbb917de3de94c80a2
I20260808 01:49:00.689123 1416 raft_consensus.cc:1275] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Refusing update from remote peer 051fc57f839541198f282afd3fcc20db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260808 01:49:00.690910 1204 consensus_peers.cc:597] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db -> Peer 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125): Couldn't send request to peer 2b4d34512c9f4851b029ef32c177d152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4578d3e670ce40bbb917de3de94c80a2. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:00.691018 1521 consensus_queue.cc:1048] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:00.701931 1522 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } } }
I20260808 01:49:00.704276 1415 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, NON_VOTER 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } } }
I20260808 01:49:00.720566 1064 consensus_queue.cc:579] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: The tablet replica hosted on peer 7b828feeb83e41289cb6db076965c7cf has failed
I20260808 01:49:00.738651 1132 consensus_queue.cc:237] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } }
I20260808 01:49:00.746788 821 catalog_manager.cc:5028] TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): tablet 4578d3e670ce40bbb917de3de94c80a2 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) successfully deleted
I20260808 01:49:00.768683 838 catalog_manager.cc:5697] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 reported cstate change: config changed from index 2 to 3, NON_VOTER 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) added. New cstate: current_term: 2 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } } }
I20260808 01:49:00.795578 821 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4578d3e670ce40bbb917de3de94c80a2 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:49:00.797158 1272 raft_consensus.cc:1275] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Refusing update from remote peer 2b4d34512c9f4851b029ef32c177d152: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:00.799263 1519 consensus_queue.cc:1048] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:00.812448 1779 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } }
I20260808 01:49:00.814575 1272 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } }
I20260808 01:49:00.824359 821 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b5b3c8b264e34085ac8839bd8668caa8 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:00.828004 837 catalog_manager.cc:5697] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 reported cstate change: config changed from index -1 to 2, VOTER 7b828feeb83e41289cb6db076965c7cf (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "2b4d34512c9f4851b029ef32c177d152" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:00.846283 1670 tablet_service.cc:1558] Processing DeleteTablet for tablet b5b3c8b264e34085ac8839bd8668caa8 with delete_type TABLET_DATA_TOMBSTONED (TS 7b828feeb83e41289cb6db076965c7cf not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:51382
I20260808 01:49:00.847452 1757 ts_tablet_manager.cc:1916] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:00.867436 1132 consensus_queue.cc:237] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } }
I20260808 01:49:00.876716 1272 raft_consensus.cc:1275] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Refusing update from remote peer 2b4d34512c9f4851b029ef32c177d152: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:00.880904 1779 consensus_queue.cc:1048] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:00.892768 1249 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260808 01:49:00.895532 1519 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } } }
I20260808 01:49:00.900952 1272 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } } }
I20260808 01:49:00.918470 821 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b5b3c8b264e34085ac8839bd8668caa8 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:49:00.916018 837 catalog_manager.cc:5697] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db reported cstate change: config changed from index 2 to 3, NON_VOTER 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "2b4d34512c9f4851b029ef32c177d152" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } } }
W20260808 01:49:00.948515 1066 consensus_peers.cc:597] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 -> Peer 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037): Couldn't send request to peer 0a462e1c7eb749fd8f898f9a85c8b691. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5b3c8b264e34085ac8839bd8668caa8. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:00.955719 1757 ts_tablet_manager.cc:1929] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:00.956205 1757 log.cc:1199] T b5b3c8b264e34085ac8839bd8668caa8 P 7b828feeb83e41289cb6db076965c7cf: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:00.965031 821 catalog_manager.cc:5028] TS 7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): tablet b5b3c8b264e34085ac8839bd8668caa8 (table test-workload [id=1ecba8e49964424c8e4c0b2852d0162f]) successfully deleted
I20260808 01:49:00.972334 1394 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:00.999567 1109 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:01.007308 1670 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:01.086673 1792 ts_tablet_manager.cc:933] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Initiating tablet copy from peer 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037)
I20260808 01:49:01.100046 1792 tablet_copy_client.cc:323] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.68:37037
I20260808 01:49:01.106009 1436 tablet_copy_service.cc:140] P 0a462e1c7eb749fd8f898f9a85c8b691: Received BeginTabletCopySession request for tablet d434eb82ce0e4369bcc673450e99f368 from peer 051fc57f839541198f282afd3fcc20db ({username='slave'} at 127.29.180.67:51471)
I20260808 01:49:01.106635 1436 tablet_copy_service.cc:161] P 0a462e1c7eb749fd8f898f9a85c8b691: Beginning new tablet copy session on tablet d434eb82ce0e4369bcc673450e99f368 from peer 051fc57f839541198f282afd3fcc20db at {username='slave'} at 127.29.180.67:51471: session id = 051fc57f839541198f282afd3fcc20db-d434eb82ce0e4369bcc673450e99f368
I20260808 01:49:01.115830 1436 tablet_copy_source_session.cc:215] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:01.169780 1792 tablet_copy_client.cc:806] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:01.170526 1792 tablet_copy_client.cc:670] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:01.176069 1792 tablet_copy_client.cc:538] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:01.189764 1794 ts_tablet_manager.cc:933] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Initiating tablet copy from peer 051fc57f839541198f282afd3fcc20db (127.29.180.67:37635)
I20260808 01:49:01.195770 1792 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Bootstrap starting.
I20260808 01:49:01.210543 1794 tablet_copy_client.cc:323] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:37635
I20260808 01:49:01.212651 1293 tablet_copy_service.cc:140] P 051fc57f839541198f282afd3fcc20db: Received BeginTabletCopySession request for tablet 4578d3e670ce40bbb917de3de94c80a2 from peer 2b4d34512c9f4851b029ef32c177d152 ({username='slave'} at 127.29.180.66:43587)
I20260808 01:49:01.213279 1293 tablet_copy_service.cc:161] P 051fc57f839541198f282afd3fcc20db: Beginning new tablet copy session on tablet 4578d3e670ce40bbb917de3de94c80a2 from peer 2b4d34512c9f4851b029ef32c177d152 at {username='slave'} at 127.29.180.66:43587: session id = 2b4d34512c9f4851b029ef32c177d152-4578d3e670ce40bbb917de3de94c80a2
I20260808 01:49:01.221503 1293 tablet_copy_source_session.cc:215] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:01.244395 1792 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:01.245493 1792 tablet_bootstrap.cc:492] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Bootstrap complete.
I20260808 01:49:01.250447 1792 ts_tablet_manager.cc:1403] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.007s
I20260808 01:49:01.253508 1792 raft_consensus.cc:359] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } }
I20260808 01:49:01.254382 1792 raft_consensus.cc:740] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 051fc57f839541198f282afd3fcc20db, State: Initialized, Role: LEARNER
I20260808 01:49:01.255071 1792 consensus_queue.cc:260] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: NON_VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: true } }
I20260808 01:49:01.269692 1792 ts_tablet_manager.cc:1434] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20260808 01:49:01.271858 1436 tablet_copy_service.cc:342] P 0a462e1c7eb749fd8f898f9a85c8b691: Request end of tablet copy session 051fc57f839541198f282afd3fcc20db-d434eb82ce0e4369bcc673450e99f368 received from {username='slave'} at 127.29.180.67:51471
I20260808 01:49:01.272390 1436 tablet_copy_service.cc:434] P 0a462e1c7eb749fd8f898f9a85c8b691: ending tablet copy session 051fc57f839541198f282afd3fcc20db-d434eb82ce0e4369bcc673450e99f368 on tablet d434eb82ce0e4369bcc673450e99f368 with peer 051fc57f839541198f282afd3fcc20db
I20260808 01:49:01.294131 1794 tablet_copy_client.cc:806] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:01.294839 1794 tablet_copy_client.cc:670] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:01.307129 1794 tablet_copy_client.cc:538] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:01.316583 1794 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap starting.
I20260808 01:49:01.318764 1797 ts_tablet_manager.cc:933] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Initiating tablet copy from peer 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125)
E20260808 01:49:01.322302 1731 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:49:01.330717 1797 tablet_copy_client.cc:323] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:36125
I20260808 01:49:01.333065 1152 tablet_copy_service.cc:140] P 2b4d34512c9f4851b029ef32c177d152: Received BeginTabletCopySession request for tablet b5b3c8b264e34085ac8839bd8668caa8 from peer 0a462e1c7eb749fd8f898f9a85c8b691 ({username='slave'} at 127.29.180.68:34995)
I20260808 01:49:01.333739 1152 tablet_copy_service.cc:161] P 2b4d34512c9f4851b029ef32c177d152: Beginning new tablet copy session on tablet b5b3c8b264e34085ac8839bd8668caa8 from peer 0a462e1c7eb749fd8f898f9a85c8b691 at {username='slave'} at 127.29.180.68:34995: session id = 0a462e1c7eb749fd8f898f9a85c8b691-b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:01.351758 1152 tablet_copy_source_session.cc:215] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:01.382766 1797 tablet_copy_client.cc:806] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:01.383422 1797 tablet_copy_client.cc:670] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:01.395092 1797 tablet_copy_client.cc:538] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:01.395412 1794 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:01.396390 1794 tablet_bootstrap.cc:492] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Bootstrap complete.
I20260808 01:49:01.397094 1794 ts_tablet_manager.cc:1403] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Time spent bootstrapping tablet: real 0.081s user 0.025s sys 0.001s
I20260808 01:49:01.399987 1794 raft_consensus.cc:359] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } }
I20260808 01:49:01.400692 1794 raft_consensus.cc:740] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2b4d34512c9f4851b029ef32c177d152, State: Initialized, Role: LEARNER
I20260808 01:49:01.401319 1794 consensus_queue.cc:260] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: true } }
I20260808 01:49:01.403918 1794 ts_tablet_manager.cc:1434] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20260808 01:49:01.406298 1293 tablet_copy_service.cc:342] P 051fc57f839541198f282afd3fcc20db: Request end of tablet copy session 2b4d34512c9f4851b029ef32c177d152-4578d3e670ce40bbb917de3de94c80a2 received from {username='slave'} at 127.29.180.66:43587
I20260808 01:49:01.406809 1293 tablet_copy_service.cc:434] P 051fc57f839541198f282afd3fcc20db: ending tablet copy session 2b4d34512c9f4851b029ef32c177d152-4578d3e670ce40bbb917de3de94c80a2 on tablet 4578d3e670ce40bbb917de3de94c80a2 with peer 2b4d34512c9f4851b029ef32c177d152
I20260808 01:49:01.418573 1797 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap starting.
I20260808 01:49:01.481333 1797 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:01.482425 1797 tablet_bootstrap.cc:492] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Bootstrap complete.
I20260808 01:49:01.483127 1797 ts_tablet_manager.cc:1403] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.012s
I20260808 01:49:01.485703 1797 raft_consensus.cc:359] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } }
I20260808 01:49:01.486421 1797 raft_consensus.cc:740] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a462e1c7eb749fd8f898f9a85c8b691, State: Initialized, Role: LEARNER
I20260808 01:49:01.487064 1797 consensus_queue.cc:260] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: true } }
I20260808 01:49:01.489221 1797 ts_tablet_manager.cc:1434] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260808 01:49:01.491786 1152 tablet_copy_service.cc:342] P 2b4d34512c9f4851b029ef32c177d152: Request end of tablet copy session 0a462e1c7eb749fd8f898f9a85c8b691-b5b3c8b264e34085ac8839bd8668caa8 received from {username='slave'} at 127.29.180.68:34995
I20260808 01:49:01.492249 1152 tablet_copy_service.cc:434] P 2b4d34512c9f4851b029ef32c177d152: ending tablet copy session 0a462e1c7eb749fd8f898f9a85c8b691-b5b3c8b264e34085ac8839bd8668caa8 on tablet b5b3c8b264e34085ac8839bd8668caa8 with peer 0a462e1c7eb749fd8f898f9a85c8b691
I20260808 01:49:01.508126 1272 raft_consensus.cc:1217] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
8285c8a80e3e4985a26f56fbac0d4b97 | 127.29.180.126:43261 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
051fc57f839541198f282afd3fcc20db | 127.29.180.67:37635 | HEALTHY | <none> | 3 | 0
0a462e1c7eb749fd8f898f9a85c8b691 | 127.29.180.68:37037 | HEALTHY | <none> | 1 | 0
2b4d34512c9f4851b029ef32c177d152 | 127.29.180.66:36125 | HEALTHY | <none> | 1 | 0
7b828feeb83e41289cb6db076965c7cf | 127.29.180.65:33489 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:33489
env_inject_eio | 1 | hidden | 127.29.180.65:33489
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:33489
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33489
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:36125
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37635
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:37037
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33489
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:36125
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37635
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:37037
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4578d3e670ce40bbb917de3de94c80a2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
051fc57f839541198f282afd3fcc20db (127.29.180.67:37635): RUNNING [LEADER]
0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037): RUNNING
All reported replicas are:
A = 051fc57f839541198f282afd3fcc20db
B = 0a462e1c7eb749fd8f898f9a85c8b691
C = 2b4d34512c9f4851b029ef32c177d152
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 2 | 3 | Yes
B | A* B C~ | 2 | 3 | Yes
Tablet b5b3c8b264e34085ac8839bd8668caa8 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125): RUNNING [LEADER]
7b828feeb83e41289cb6db076965c7cf (127.29.180.65:33489): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
051fc57f839541198f282afd3fcc20db (127.29.180.67:37635): RUNNING
All reported replicas are:
A = 2b4d34512c9f4851b029ef32c177d152
B = 7b828feeb83e41289cb6db076965c7cf
C = 051fc57f839541198f282afd3fcc20db
D = 0a462e1c7eb749fd8f898f9a85c8b691
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* C D~ | 1 | 3 | Yes
B | [config not available] | | |
C | A* C D~ | 1 | 3 | No
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 3
Median | 4
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 14
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:49:01.604568 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:49:01.655417 1132 raft_consensus.cc:1217] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 01:49:01.821182 1415 raft_consensus.cc:1217] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260808 01:49:02.034835 1825 raft_consensus.cc:1064] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691: attempting to promote NON_VOTER 051fc57f839541198f282afd3fcc20db to VOTER
I20260808 01:49:02.036881 1825 consensus_queue.cc:237] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false } }
I20260808 01:49:02.045531 1272 raft_consensus.cc:1275] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [term 1 LEARNER]: Refusing update from remote peer 0a462e1c7eb749fd8f898f9a85c8b691: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:02.047858 1132 raft_consensus.cc:1275] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Refusing update from remote peer 0a462e1c7eb749fd8f898f9a85c8b691: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:02.048688 1749 consensus_queue.cc:1048] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:02.050274 1827 consensus_queue.cc:1048] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:02.061381 1825 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 051fc57f839541198f282afd3fcc20db (127.29.180.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false } } }
I20260808 01:49:02.063279 1132 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 051fc57f839541198f282afd3fcc20db (127.29.180.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false } } }
I20260808 01:49:02.066884 1271 raft_consensus.cc:2955] T d434eb82ce0e4369bcc673450e99f368 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 051fc57f839541198f282afd3fcc20db (127.29.180.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false } } }
I20260808 01:49:02.078374 836 catalog_manager.cc:5697] T d434eb82ce0e4369bcc673450e99f368 P 2b4d34512c9f4851b029ef32c177d152 reported cstate change: config changed from index 3 to 4, 051fc57f839541198f282afd3fcc20db (127.29.180.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } attrs { promote: false } } }
I20260808 01:49:02.213065 1394 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:02.234805 1670 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:02.268630 1109 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:02.284021 1848 raft_consensus.cc:1064] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db: attempting to promote NON_VOTER 2b4d34512c9f4851b029ef32c177d152 to VOTER
W20260808 01:49:02.286131 1206 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.68:37037, user_credentials={real_user=slave}} blocked reactor thread for 61500.3us
I20260808 01:49:02.311398 1848 consensus_queue.cc:237] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false } }
W20260808 01:49:02.313750 1204 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:36125, user_credentials={real_user=slave}} blocked reactor thread for 95343.5us
I20260808 01:49:02.332386 1415 raft_consensus.cc:1275] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Refusing update from remote peer 051fc57f839541198f282afd3fcc20db: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:02.337975 1848 consensus_queue.cc:1048] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:02.344424 1132 raft_consensus.cc:1275] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [term 2 LEARNER]: Refusing update from remote peer 051fc57f839541198f282afd3fcc20db: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:02.350278 1776 consensus_queue.cc:1048] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:02.362166 1249 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260808 01:49:02.364679 1845 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false } } }
I20260808 01:49:02.366513 1415 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false } } }
I20260808 01:49:02.379747 1132 raft_consensus.cc:2955] T 4578d3e670ce40bbb917de3de94c80a2 P 2b4d34512c9f4851b029ef32c177d152 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false } } }
I20260808 01:49:02.391628 838 catalog_manager.cc:5697] T 4578d3e670ce40bbb917de3de94c80a2 P 051fc57f839541198f282afd3fcc20db reported cstate change: config changed from index 3 to 4, 2b4d34512c9f4851b029ef32c177d152 (127.29.180.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "051fc57f839541198f282afd3fcc20db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:02.454733 1813 raft_consensus.cc:1064] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152: attempting to promote NON_VOTER 0a462e1c7eb749fd8f898f9a85c8b691 to VOTER
I20260808 01:49:02.457022 1813 consensus_queue.cc:237] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false } }
I20260808 01:49:02.494661 1415 raft_consensus.cc:1275] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 LEARNER]: Refusing update from remote peer 2b4d34512c9f4851b029ef32c177d152: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:02.495639 1271 raft_consensus.cc:1275] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Refusing update from remote peer 2b4d34512c9f4851b029ef32c177d152: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:02.498116 1853 consensus_queue.cc:1048] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:02.499958 1853 consensus_queue.cc:1048] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:02.535480 1272 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false } } }
I20260808 01:49:02.541911 1813 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 2b4d34512c9f4851b029ef32c177d152 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false } } }
I20260808 01:49:02.553583 838 catalog_manager.cc:5697] T b5b3c8b264e34085ac8839bd8668caa8 P 051fc57f839541198f282afd3fcc20db reported cstate change: config changed from index 3 to 4, 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2b4d34512c9f4851b029ef32c177d152" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false } } }
I20260808 01:49:02.578251 1415 raft_consensus.cc:2955] T b5b3c8b264e34085ac8839bd8668caa8 P 0a462e1c7eb749fd8f898f9a85c8b691 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2b4d34512c9f4851b029ef32c177d152" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 36125 } } peers { permanent_uuid: "051fc57f839541198f282afd3fcc20db" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 37635 } } peers { permanent_uuid: "0a462e1c7eb749fd8f898f9a85c8b691" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 37037 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
8285c8a80e3e4985a26f56fbac0d4b97 | 127.29.180.126:43261 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
051fc57f839541198f282afd3fcc20db | 127.29.180.67:37635 | HEALTHY | <none> | 3 | 0
0a462e1c7eb749fd8f898f9a85c8b691 | 127.29.180.68:37037 | HEALTHY | <none> | 1 | 0
2b4d34512c9f4851b029ef32c177d152 | 127.29.180.66:36125 | HEALTHY | <none> | 1 | 0
7b828feeb83e41289cb6db076965c7cf | 127.29.180.65:33489 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:33489
env_inject_eio | 1 | hidden | 127.29.180.65:33489
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:33489
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33489
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:36125
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37635
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:37037
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33489
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:36125
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37635
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:37037
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 4578d3e670ce40bbb917de3de94c80a2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
051fc57f839541198f282afd3fcc20db (127.29.180.67:37635): RUNNING [LEADER]
0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037): RUNNING
2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125): RUNNING [NONVOTER]
All reported replicas are:
A = 051fc57f839541198f282afd3fcc20db
B = 0a462e1c7eb749fd8f898f9a85c8b691
C = 2b4d34512c9f4851b029ef32c177d152
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C | 2 | 4 | Yes
B | A* B C~ | 2 | 3 | Yes
C | A* B C~ | 2 | 3 | Yes
Tablet d434eb82ce0e4369bcc673450e99f368 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's
2b4d34512c9f4851b029ef32c177d152 (127.29.180.66:36125): RUNNING
0a462e1c7eb749fd8f898f9a85c8b691 (127.29.180.68:37037): RUNNING [LEADER]
051fc57f839541198f282afd3fcc20db (127.29.180.67:37635): RUNNING [NONVOTER]
All reported replicas are:
A = 2b4d34512c9f4851b029ef32c177d152
B = 0a462e1c7eb749fd8f898f9a85c8b691
C = 051fc57f839541198f282afd3fcc20db
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C | 1 | 4 | Yes
B | A B* C | 1 | 4 | Yes
C | A B* C | 1 | 4 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 0 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 7b828feeb83e41289cb6db076965c7cf | 127.29.180.65:33489 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:49:02.840938 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:49:03.365170 1249 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260808 01:49:03.373351 1394 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:03.398290 1109 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:03.405462 1670 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
8285c8a80e3e4985a26f56fbac0d4b97 | 127.29.180.126:43261 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
051fc57f839541198f282afd3fcc20db | 127.29.180.67:37635 | HEALTHY | <none> | 3 | 0
0a462e1c7eb749fd8f898f9a85c8b691 | 127.29.180.68:37037 | HEALTHY | <none> | 1 | 0
2b4d34512c9f4851b029ef32c177d152 | 127.29.180.66:36125 | HEALTHY | <none> | 1 | 0
7b828feeb83e41289cb6db076965c7cf | 127.29.180.65:33489 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:33489
env_inject_eio | 1 | hidden | 127.29.180.65:33489
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:33489
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33489
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:36125
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:37635
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:37037
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33489
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:36125
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:37635
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:37037
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:46031 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 7b828feeb83e41289cb6db076965c7cf | 127.29.180.65:33489 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:49:03.690173 30417 log_verifier.cc:126] Checking tablet 335fdcca02cd4457b5e220acc4c9bb08
I20260808 01:49:03.758973 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 335fdcca02cd4457b5e220acc4c9bb08
I20260808 01:49:03.759271 30417 log_verifier.cc:126] Checking tablet 369b6f293ec74f32b02e936e6539461a
I20260808 01:49:03.843967 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 369b6f293ec74f32b02e936e6539461a
I20260808 01:49:03.844172 30417 log_verifier.cc:126] Checking tablet 4578d3e670ce40bbb917de3de94c80a2
I20260808 01:49:03.909556 30417 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 4578d3e670ce40bbb917de3de94c80a2
I20260808 01:49:03.909868 30417 log_verifier.cc:126] Checking tablet b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:04.014495 30417 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b5b3c8b264e34085ac8839bd8668caa8
I20260808 01:49:04.014721 30417 log_verifier.cc:126] Checking tablet d434eb82ce0e4369bcc673450e99f368
I20260808 01:49:04.113538 30417 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d434eb82ce0e4369bcc673450e99f368
I20260808 01:49:04.657927 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 1560
I20260808 01:49:04.715610 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 1041
I20260808 01:49:04.787569 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 1181
I20260808 01:49:04.840749 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 1323
I20260808 01:49:04.881505 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 797
2026-08-08T01:49:04Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3 (21295 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/4
2026-08-08T01:49:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:49:04Z Disabled control of system clock
I20260808 01:49:05.017199 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41533
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:41533
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:49:05.344601 1909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:05.345206 1909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:05.345597 1909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:05.379824 1909 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:49:05.380216 1909 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:49:05.380426 1909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:05.380627 1909 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:49:05.380825 1909 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:49:05.420585 1909 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:41533
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:05.422014 1909 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:05.423722 1909 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:05.436025 1915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:05.436674 1916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:05.447026 1918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:06.681841 1917 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1241 milliseconds
I20260808 01:49:06.681978 1909 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:06.683449 1909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:06.686254 1909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:06.687673 1909 hybrid_clock.cc:648] HybridClock initialized: now 1786153746687636 us; error 77 us; skew 500 ppm
I20260808 01:49:06.688633 1909 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:06.696135 1909 webserver.cc:533] Webserver started at http://127.29.180.126:36475/ using document root <none> and password file <none>
I20260808 01:49:06.697845 1909 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:06.698119 1909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:06.698904 1909 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:06.703693 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:06.704937 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:06.705915 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:06.706864 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:06.707777 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:06.708772 1909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:07.316367 1909 fs_manager.cc:696] Time spent creating directory manager: real 0.607s user 0.257s sys 0.259s
I20260808 01:49:07.326691 1909 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260808 01:49:07.327121 1909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "079165bb7d084b7d9222554b096fbd3e"
format_stamp: "Formatted at 2026-08-08 01:49:06 on dist-test-slave-p3t5"
I20260808 01:49:07.327529 1909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:07.382318 1909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:07.383888 1909 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:07.384440 1909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:07.460558 1909 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:41533
I20260808 01:49:07.460680 2067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:41533 every 8 connection(s)
I20260808 01:49:07.463558 1909 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:49:07.471174 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 1909
I20260808 01:49:07.471768 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:49:07.492583 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e: Bootstrap starting.
I20260808 01:49:07.498689 2068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.500815 2068 log.cc:826] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:07.506337 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e: No bootstrap required, opened a new log
I20260808 01:49:07.526890 2068 raft_consensus.cc:359] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } }
I20260808 01:49:07.527899 2068 raft_consensus.cc:385] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.528172 2068 raft_consensus.cc:740] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079165bb7d084b7d9222554b096fbd3e, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.529057 2068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } }
I20260808 01:49:07.529627 2068 raft_consensus.cc:399] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:49:07.529981 2068 raft_consensus.cc:493] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:49:07.530344 2068 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.534832 2068 raft_consensus.cc:515] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } }
I20260808 01:49:07.535910 2068 leader_election.cc:304] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 079165bb7d084b7d9222554b096fbd3e; no voters:
I20260808 01:49:07.538120 2068 leader_election.cc:290] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:49:07.538630 2073 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.541296 2073 raft_consensus.cc:697] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [term 1 LEADER]: Becoming Leader. State: Replica: 079165bb7d084b7d9222554b096fbd3e, State: Running, Role: LEADER
I20260808 01:49:07.542436 2073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } }
I20260808 01:49:07.543555 2068 sys_catalog.cc:565] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:49:07.585079 2074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "079165bb7d084b7d9222554b096fbd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } } }
I20260808 01:49:07.585510 2075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 079165bb7d084b7d9222554b096fbd3e. Latest consensus state: current_term: 1 leader_uuid: "079165bb7d084b7d9222554b096fbd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079165bb7d084b7d9222554b096fbd3e" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 41533 } } }
I20260808 01:49:07.585983 2074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:07.586153 2075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:07.589535 2082 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:49:07.602505 2082 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:49:07.620031 2082 catalog_manager.cc:1383] Generated new cluster ID: 4f115599aa3e491b8932a04a00c0758b
I20260808 01:49:07.620348 2082 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:49:07.640604 2082 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:49:07.642256 2082 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:49:07.656605 2082 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 079165bb7d084b7d9222554b096fbd3e: Generated new TSK 0
I20260808 01:49:07.657660 2082 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:49:07.676491 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--builtin_ntp_servers=127.29.180.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20260808 01:49:08.008762 2093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:08.009358 2093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:08.009927 2093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:08.044009 2093 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:49:08.044471 2093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:08.045291 2093 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:49:08.083946 2093 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:08.085443 2093 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:08.087267 2093 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:08.101008 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:08.102136 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:09.503153 2098 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 2093
W20260808 01:49:09.637053 2098 kernel_stack_watchdog.cc:198] Thread 2093 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:49:09.639271 2093 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.536s user 0.000s sys 0.002s
W20260808 01:49:09.639736 2093 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.537s user 0.000s sys 0.003s
W20260808 01:49:09.641826 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:09.645979 2101 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1542 milliseconds
I20260808 01:49:09.646052 2093 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:09.647373 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:09.650141 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:09.651635 2093 hybrid_clock.cc:648] HybridClock initialized: now 1786153749651591 us; error 53 us; skew 500 ppm
I20260808 01:49:09.652515 2093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:09.659490 2093 webserver.cc:533] Webserver started at http://127.29.180.65:39357/ using document root <none> and password file <none>
I20260808 01:49:09.661008 2093 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:09.661284 2093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:09.662045 2093 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:09.666589 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:09.667938 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:09.669036 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:09.670058 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:09.670971 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:09.671882 2093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:10.284282 2093 fs_manager.cc:696] Time spent creating directory manager: real 0.612s user 0.366s sys 0.166s
I20260808 01:49:10.294190 2093 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.002s
I20260808 01:49:10.294575 2093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:10.294961 2093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:10.359053 2093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:10.360522 2093 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:10.361007 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:10.363152 2093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:10.366904 2093 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:10.367130 2093 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:10.367404 2093 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:10.367570 2093 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:10.506769 2093 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:37873
I20260808 01:49:10.506880 2317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:37873 every 8 connection(s)
I20260808 01:49:10.509407 2093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:49:10.510218 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 2093
I20260808 01:49:10.511039 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:49:10.519629 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--builtin_ntp_servers=127.29.180.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:49:10.564447 2318 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41533
I20260808 01:49:10.564980 2318 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:10.566416 2318 heartbeater.cc:507] Master 127.29.180.126:41533 requested a full tablet report, sending...
I20260808 01:49:10.569835 2033 ts_manager.cc:194] Registered new tserver with Master: 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873)
I20260808 01:49:10.571904 2033 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:38695
W20260808 01:49:10.854554 2322 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:10.855168 2322 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:10.855706 2322 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:10.889778 2322 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:49:10.890271 2322 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:10.891105 2322 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:49:10.930696 2322 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:10.932261 2322 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:10.934147 2322 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:10.947094 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:11.575557 2318 heartbeater.cc:499] Master 127.29.180.126:41533 was elected leader, sending a full tablet report...
W20260808 01:49:10.947746 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:12.271199 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:12.273590 2330 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1321 milliseconds
I20260808 01:49:12.273693 2322 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:12.274992 2322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:12.277817 2322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:12.279282 2322 hybrid_clock.cc:648] HybridClock initialized: now 1786153752279226 us; error 67 us; skew 500 ppm
I20260808 01:49:12.280229 2322 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:12.287150 2322 webserver.cc:533] Webserver started at http://127.29.180.66:34471/ using document root <none> and password file <none>
I20260808 01:49:12.288715 2322 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:12.288982 2322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:12.289746 2322 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:12.294369 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.295565 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.296494 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.297363 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.298266 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.299144 2322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.889585 2322 fs_manager.cc:696] Time spent creating directory manager: real 0.590s user 0.221s sys 0.285s
I20260808 01:49:12.898948 2322 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260808 01:49:12.899317 2322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "4e46a7579cc348569736e0e431dbddb6"
format_stamp: "Formatted at 2026-08-08 01:49:12 on dist-test-slave-p3t5"
I20260808 01:49:12.899712 2322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:12.958125 2322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:12.959575 2322 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:12.960058 2322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:12.962219 2322 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:12.966020 2322 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:12.966261 2322 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:12.966518 2322 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:12.966694 2322 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:13.104233 2322 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:42617
I20260808 01:49:13.104357 2544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:42617 every 8 connection(s)
I20260808 01:49:13.106817 2322 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:49:13.110437 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 2322
I20260808 01:49:13.111027 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:49:13.118651 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--builtin_ntp_servers=127.29.180.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:49:13.134317 2545 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41533
I20260808 01:49:13.134822 2545 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:13.135921 2545 heartbeater.cc:507] Master 127.29.180.126:41533 requested a full tablet report, sending...
I20260808 01:49:13.138223 2033 ts_manager.cc:194] Registered new tserver with Master: 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
I20260808 01:49:13.139624 2033 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:51571
W20260808 01:49:13.446209 2549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:13.446835 2549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:13.447347 2549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:13.492480 2549 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:49:13.493076 2549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:13.494009 2549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:49:13.533696 2549 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:41533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:13.535246 2549 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:13.537153 2549 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:13.551019 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:14.143034 2545 heartbeater.cc:499] Master 127.29.180.126:41533 was elected leader, sending a full tablet report...
W20260808 01:49:13.552227 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:14.756451 2557 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260808 01:49:14.758061 2558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:14.758073 2549 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:14.761294 2549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:14.764297 2549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:14.765746 2549 hybrid_clock.cc:648] HybridClock initialized: now 1786153754765707 us; error 50 us; skew 500 ppm
I20260808 01:49:14.766654 2549 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:14.773337 2549 webserver.cc:533] Webserver started at http://127.29.180.67:38717/ using document root <none> and password file <none>
I20260808 01:49:14.774938 2549 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:14.775183 2549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:14.775887 2549 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:14.780742 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:14.782083 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:14.783020 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:14.783919 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:14.784893 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:14.785821 2549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:15.414590 2549 fs_manager.cc:696] Time spent creating directory manager: real 0.628s user 0.250s sys 0.290s
I20260808 01:49:15.424544 2549 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260808 01:49:15.424860 2549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
format_stamp: "Formatted at 2026-08-08 01:49:14 on dist-test-slave-p3t5"
I20260808 01:49:15.425199 2549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:15.485735 2549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:15.487140 2549 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:15.487742 2549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:15.490404 2549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:15.494184 2549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:15.494376 2549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:15.494568 2549 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:15.494690 2549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:15.635660 2549 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:42539
I20260808 01:49:15.635864 2772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:42539 every 8 connection(s)
I20260808 01:49:15.638368 2549 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:49:15.647404 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 2549
I20260808 01:49:15.647876 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:49:15.660874 2773 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41533
I20260808 01:49:15.661298 2773 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:15.662381 2773 heartbeater.cc:507] Master 127.29.180.126:41533 requested a full tablet report, sending...
I20260808 01:49:15.664677 2033 ts_manager.cc:194] Registered new tserver with Master: e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:15.666530 2033 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:38605
I20260808 01:49:15.670013 30417 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260808 01:49:15.748952 2033 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39460:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:49:15.849565 2708 tablet_service.cc:1511] Processing CreateTablet for tablet b88d1c1afe054fd3b0557b47c5238cf8 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:15.852459 2707 tablet_service.cc:1511] Processing CreateTablet for tablet d4c031377aec49afaa707e2cbeb840d2 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:15.854311 2706 tablet_service.cc:1511] Processing CreateTablet for tablet ef96d18afb024e62a17e2c0f0f035c5b (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:15.854328 2477 tablet_service.cc:1511] Processing CreateTablet for tablet d4c031377aec49afaa707e2cbeb840d2 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:15.858140 2475 tablet_service.cc:1511] Processing CreateTablet for tablet fa85170369884692b5cf1abe2fc79afd (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:15.858505 2705 tablet_service.cc:1511] Processing CreateTablet for tablet fa85170369884692b5cf1abe2fc79afd (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:15.854059 2478 tablet_service.cc:1511] Processing CreateTablet for tablet b88d1c1afe054fd3b0557b47c5238cf8 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:15.849424 2704 tablet_service.cc:1511] Processing CreateTablet for tablet a0cc3757fca846a189aae3a943265a08 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:15.862066 2476 tablet_service.cc:1511] Processing CreateTablet for tablet ef96d18afb024e62a17e2c0f0f035c5b (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:15.860342 2251 tablet_service.cc:1511] Processing CreateTablet for tablet b88d1c1afe054fd3b0557b47c5238cf8 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:15.864207 2474 tablet_service.cc:1511] Processing CreateTablet for tablet a0cc3757fca846a189aae3a943265a08 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:15.866460 2250 tablet_service.cc:1511] Processing CreateTablet for tablet d4c031377aec49afaa707e2cbeb840d2 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:15.866858 2249 tablet_service.cc:1511] Processing CreateTablet for tablet ef96d18afb024e62a17e2c0f0f035c5b (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:15.879930 2248 tablet_service.cc:1511] Processing CreateTablet for tablet fa85170369884692b5cf1abe2fc79afd (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:15.887377 2247 tablet_service.cc:1511] Processing CreateTablet for tablet a0cc3757fca846a189aae3a943265a08 (DEFAULT_TABLE table=test-workload [id=4e15842e93314742887b2318731f2757]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:15.904398 2798 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6: Bootstrap starting.
I20260808 01:49:15.945674 2799 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:15.950578 2803 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:15.951651 2804 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:15.961741 2799 tablet_bootstrap.cc:654] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.952862 2802 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:15.962267 2804 tablet_bootstrap.cc:654] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.940834 2800 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: Bootstrap starting.
I20260808 01:49:15.953914 2805 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: Bootstrap starting.
I20260808 01:49:15.971983 2804 log.cc:826] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:15.975046 2800 tablet_bootstrap.cc:654] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.974133 2802 tablet_bootstrap.cc:654] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.977883 2800 log.cc:826] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:15.983418 2798 tablet_bootstrap.cc:654] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.989090 2808 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Bootstrap starting.
I20260808 01:49:15.990214 2806 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: Bootstrap starting.
I20260808 01:49:15.990338 2805 tablet_bootstrap.cc:654] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.991572 2800 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: No bootstrap required, opened a new log
I20260808 01:49:15.992237 2800 ts_tablet_manager.cc:1403] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.015s
I20260808 01:49:15.999972 2803 tablet_bootstrap.cc:654] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.001873 2811 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:16.012132 2809 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d: Bootstrap starting.
I20260808 01:49:16.012506 2802 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: No bootstrap required, opened a new log
I20260808 01:49:16.013115 2802 ts_tablet_manager.cc:1403] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.006s
I20260808 01:49:16.016921 2810 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d: Bootstrap starting.
I20260808 01:49:16.019093 2816 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: Bootstrap starting.
I20260808 01:49:16.021435 2798 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6: No bootstrap required, opened a new log
I20260808 01:49:16.022071 2798 ts_tablet_manager.cc:1403] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.013s
I20260808 01:49:16.026304 2804 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: No bootstrap required, opened a new log
I20260808 01:49:16.027047 2804 ts_tablet_manager.cc:1403] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.003s
I20260808 01:49:16.028726 2819 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d: Bootstrap starting.
I20260808 01:49:16.033042 2799 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: No bootstrap required, opened a new log
I20260808 01:49:16.033743 2799 ts_tablet_manager.cc:1403] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.010s
I20260808 01:49:16.037449 2811 tablet_bootstrap.cc:654] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.044912 2807 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Bootstrap starting.
I20260808 01:49:16.048743 2808 tablet_bootstrap.cc:654] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.050035 2805 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: No bootstrap required, opened a new log
I20260808 01:49:16.050486 2805 ts_tablet_manager.cc:1403] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.006s
I20260808 01:49:16.053300 2805 raft_consensus.cc:359] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.053160 2800 raft_consensus.cc:359] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.054349 2800 raft_consensus.cc:385] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.054647 2800 raft_consensus.cc:740] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.054608 2798 raft_consensus.cc:359] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.055135 2811 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: No bootstrap required, opened a new log
I20260808 01:49:16.055337 2798 raft_consensus.cc:385] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.055552 2811 ts_tablet_manager.cc:1403] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.000s
I20260808 01:49:16.055615 2798 raft_consensus.cc:740] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.055732 2803 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: No bootstrap required, opened a new log
I20260808 01:49:16.056119 2803 ts_tablet_manager.cc:1403] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.010s
I20260808 01:49:16.056360 2798 consensus_queue.cc:260] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.055712 2800 consensus_queue.cc:260] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.057083 2805 raft_consensus.cc:385] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.057425 2805 raft_consensus.cc:740] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.058486 2805 consensus_queue.cc:260] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.062359 2800 ts_tablet_manager.cc:1434] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6: Time spent starting tablet: real 0.070s user 0.023s sys 0.000s
I20260808 01:49:16.066327 2806 tablet_bootstrap.cc:654] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.078441 2803 raft_consensus.cc:359] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.080828 2803 raft_consensus.cc:385] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.080839 2816 tablet_bootstrap.cc:654] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.081257 2803 raft_consensus.cc:740] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.080006 2811 raft_consensus.cc:359] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.082350 2811 raft_consensus.cc:385] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.082736 2811 raft_consensus.cc:740] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.083691 2811 consensus_queue.cc:260] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.084887 2803 consensus_queue.cc:260] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.086288 2808 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: No bootstrap required, opened a new log
I20260808 01:49:16.086859 2808 ts_tablet_manager.cc:1403] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.004s
I20260808 01:49:16.087044 2802 raft_consensus.cc:359] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.089725 2802 raft_consensus.cc:385] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.090058 2802 raft_consensus.cc:740] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.091051 2808 raft_consensus.cc:359] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.091857 2799 raft_consensus.cc:359] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.096971 2799 raft_consensus.cc:385] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.097344 2799 raft_consensus.cc:740] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.093890 2804 raft_consensus.cc:359] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.098150 2804 raft_consensus.cc:385] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.099484 2804 raft_consensus.cc:740] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.100517 2804 consensus_queue.cc:260] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.098593 2805 ts_tablet_manager.cc:1434] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: Time spent starting tablet: real 0.048s user 0.009s sys 0.000s
I20260808 01:49:16.103708 2798 ts_tablet_manager.cc:1434] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6: Time spent starting tablet: real 0.081s user 0.025s sys 0.036s
I20260808 01:49:16.105645 2807 tablet_bootstrap.cc:654] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.108839 2806 log.cc:826] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: Log is configured to *not* fsync() on all Append() calls
W20260808 01:49:16.114864 2546 tablet.cc:2406] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:16.101411 2808 raft_consensus.cc:385] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.120709 2808 raft_consensus.cc:740] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.095765 2811 ts_tablet_manager.cc:1434] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.040s user 0.010s sys 0.000s
I20260808 01:49:16.096213 2802 consensus_queue.cc:260] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.098640 2799 consensus_queue.cc:260] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.114992 2803 ts_tablet_manager.cc:1434] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.059s user 0.007s sys 0.003s
I20260808 01:49:16.117815 2819 tablet_bootstrap.cc:654] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.126489 2824 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.127415 2824 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.138792 2826 raft_consensus.cc:493] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.143487 2804 ts_tablet_manager.cc:1434] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.116s user 0.015s sys 0.005s
I20260808 01:49:16.145745 2809 tablet_bootstrap.cc:654] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.151226 2810 tablet_bootstrap.cc:654] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:16.139564 2826 raft_consensus.cc:515] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.156428 2802 ts_tablet_manager.cc:1434] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.143s user 0.018s sys 0.009s
I20260808 01:49:16.157936 2829 raft_consensus.cc:493] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.161454 2806 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: No bootstrap required, opened a new log
I20260808 01:49:16.161957 2829 raft_consensus.cc:515] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.166574 2806 ts_tablet_manager.cc:1403] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: Time spent bootstrapping tablet: real 0.178s user 0.017s sys 0.004s
I20260808 01:49:16.121536 2808 consensus_queue.cc:260] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.169659 2799 ts_tablet_manager.cc:1434] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.135s user 0.019s sys 0.006s
I20260808 01:49:16.176546 2808 ts_tablet_manager.cc:1434] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Time spent starting tablet: real 0.089s user 0.004s sys 0.002s
I20260808 01:49:16.184262 2816 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: No bootstrap required, opened a new log
I20260808 01:49:16.187491 2825 raft_consensus.cc:493] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.188400 2825 raft_consensus.cc:515] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.189555 2829 leader_election.cc:290] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.190826 2826 leader_election.cc:290] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.199319 2825 leader_election.cc:290] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.201995 2816 ts_tablet_manager.cc:1403] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent bootstrapping tablet: real 0.183s user 0.023s sys 0.000s
I20260808 01:49:16.214546 2824 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.221890 2807 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: No bootstrap required, opened a new log
I20260808 01:49:16.222406 2807 ts_tablet_manager.cc:1403] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Time spent bootstrapping tablet: real 0.178s user 0.038s sys 0.078s
I20260808 01:49:16.223204 2810 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d: No bootstrap required, opened a new log
I20260808 01:49:16.227816 2807 raft_consensus.cc:359] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.224251 2819 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d: No bootstrap required, opened a new log
I20260808 01:49:16.228600 2807 raft_consensus.cc:385] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.228915 2807 raft_consensus.cc:740] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.229003 2819 ts_tablet_manager.cc:1403] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d: Time spent bootstrapping tablet: real 0.201s user 0.016s sys 0.004s
I20260808 01:49:16.229674 2807 consensus_queue.cc:260] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.231814 2807 ts_tablet_manager.cc:1434] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260808 01:49:16.228780 2816 raft_consensus.cc:359] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.236976 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:16.237605 2498 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:16.238368 2498 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.237797 2499 raft_consensus.cc:2468] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.240314 2207 leader_election.cc:304] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.235184 2806 raft_consensus.cc:359] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.240998 2806 raft_consensus.cc:385] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.241343 2806 raft_consensus.cc:740] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e7a0a64b46568c6c3c97e9fc412d, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.242185 2829 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.241407 2826 raft_consensus.cc:2804] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.242755 2826 raft_consensus.cc:493] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.242430 2806 consensus_queue.cc:260] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.242633 2829 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.243131 2826 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.244710 2829 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.247898 2207 leader_election.cc:304] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.235517 2810 ts_tablet_manager.cc:1403] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.006s
I20260808 01:49:16.235230 2816 raft_consensus.cc:385] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.249532 2816 raft_consensus.cc:740] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e7a0a64b46568c6c3c97e9fc412d, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.251977 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:16.252796 2499 raft_consensus.cc:2468] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.254729 2207 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.255560 2272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085f84d6c0e44b1995bafb3d7a969cca" is_pre_election: true
I20260808 01:49:16.256670 2272 raft_consensus.cc:2468] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 0.
I20260808 01:49:16.256702 2830 raft_consensus.cc:2804] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.257506 2830 raft_consensus.cc:493] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.258093 2830 raft_consensus.cc:3060] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.258977 2271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0cc3757fca846a189aae3a943265a08" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085f84d6c0e44b1995bafb3d7a969cca" is_pre_election: true
I20260808 01:49:16.259999 2271 raft_consensus.cc:2468] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 0.
I20260808 01:49:16.261649 2432 leader_election.cc:304] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.262746 2829 raft_consensus.cc:2804] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.263093 2829 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.264313 2432 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.263406 2829 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.265184 2825 raft_consensus.cc:2804] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.265663 2825 raft_consensus.cc:493] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.266060 2825 raft_consensus.cc:3060] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.267095 2816 consensus_queue.cc:260] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.271468 2826 raft_consensus.cc:515] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.273164 2819 raft_consensus.cc:359] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
W20260808 01:49:16.273743 2319 tablet.cc:2406] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:16.274115 2819 raft_consensus.cc:385] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.274405 2819 raft_consensus.cc:740] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e7a0a64b46568c6c3c97e9fc412d, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.277793 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6"
I20260808 01:49:16.278604 2499 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.278410 2728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.279003 2829 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.279407 2728 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.281174 2829 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Requested vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.281203 2726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.281785 2826 leader_election.cc:290] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Requested vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.283095 2498 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6"
I20260808 01:49:16.283692 2498 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:16.287680 2206 leader_election.cc:343] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:16.285777 2727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.290488 2726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.291791 2728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.292372 2728 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:16.294663 2206 leader_election.cc:343] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:16.295943 2498 raft_consensus.cc:2468] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 1.
I20260808 01:49:16.298714 2207 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters: e910e7a0a64b46568c6c3c97e9fc412d
I20260808 01:49:16.300105 2829 raft_consensus.cc:2804] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:16.300607 2829 raft_consensus.cc:697] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 LEADER]: Becoming Leader. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Running, Role: LEADER
I20260808 01:49:16.301640 2829 consensus_queue.cc:237] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.302181 2825 raft_consensus.cc:515] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.305112 2825 leader_election.cc:290] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 election: Requested vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.296384 2830 raft_consensus.cc:515] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.306885 2272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0cc3757fca846a189aae3a943265a08" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085f84d6c0e44b1995bafb3d7a969cca"
I20260808 01:49:16.307621 2272 raft_consensus.cc:3060] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.308020 2824 raft_consensus.cc:2764] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:49:16.310300 2497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6"
I20260808 01:49:16.311574 2497 raft_consensus.cc:3060] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.315706 2272 raft_consensus.cc:2468] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:16.318831 2432 leader_election.cc:304] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.309151 2830 leader_election.cc:290] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Requested vote from peers 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:16.313217 2499 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 1.
I20260808 01:49:16.307773 2810 raft_consensus.cc:359] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.321552 2497 raft_consensus.cc:2468] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 1.
I20260808 01:49:16.323351 2824 raft_consensus.cc:2804] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:16.323083 2207 leader_election.cc:304] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.324195 2824 raft_consensus.cc:697] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Becoming Leader. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Running, Role: LEADER
I20260808 01:49:16.324715 2207 leader_election.cc:304] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.325398 2829 raft_consensus.cc:2804] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:16.325723 2725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.325384 2824 consensus_queue.cc:237] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.326387 2829 raft_consensus.cc:697] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 LEADER]: Becoming Leader. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Running, Role: LEADER
I20260808 01:49:16.327342 2829 consensus_queue.cc:237] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.327688 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0cc3757fca846a189aae3a943265a08" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.328368 2724 raft_consensus.cc:3060] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.328652 2826 raft_consensus.cc:2804] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:16.329141 2826 raft_consensus.cc:697] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 1 LEADER]: Becoming Leader. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Running, Role: LEADER
I20260808 01:49:16.330076 2826 consensus_queue.cc:237] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
W20260808 01:49:16.335045 2433 leader_election.cc:343] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:16.324097 2810 raft_consensus.cc:385] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.335088 2723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.336711 2810 raft_consensus.cc:740] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e7a0a64b46568c6c3c97e9fc412d, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.337571 2033 catalog_manager.cc:5697] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca reported cstate change: term changed from 0 to 1, leader changed from <none> to 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65). New cstate: current_term: 1 leader_uuid: "085f84d6c0e44b1995bafb3d7a969cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:16.339380 2810 consensus_queue.cc:260] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.339777 2728 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 1.
I20260808 01:49:16.340723 2726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0cc3757fca846a189aae3a943265a08" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.343219 2809 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d: No bootstrap required, opened a new log
I20260808 01:49:16.343698 2809 ts_tablet_manager.cc:1403] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent bootstrapping tablet: real 0.332s user 0.013s sys 0.014s
I20260808 01:49:16.345582 2773 heartbeater.cc:499] Master 127.29.180.126:41533 was elected leader, sending a full tablet report...
I20260808 01:49:16.345474 2032 catalog_manager.cc:5697] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:16.348627 2724 raft_consensus.cc:2468] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:16.351542 2809 raft_consensus.cc:359] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.352459 2809 raft_consensus.cc:385] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:16.352846 2809 raft_consensus.cc:740] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e910e7a0a64b46568c6c3c97e9fc412d, State: Initialized, Role: FOLLOWER
I20260808 01:49:16.353150 2806 ts_tablet_manager.cc:1434] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d: Time spent starting tablet: real 0.186s user 0.019s sys 0.015s
I20260808 01:49:16.353945 2809 consensus_queue.cc:260] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.355523 2816 ts_tablet_manager.cc:1434] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent starting tablet: real 0.153s user 0.022s sys 0.027s
I20260808 01:49:16.356982 2809 ts_tablet_manager.cc:1434] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260808 01:49:16.364554 2819 consensus_queue.cc:260] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.365330 2810 ts_tablet_manager.cc:1434] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d: Time spent starting tablet: real 0.116s user 0.009s sys 0.004s
I20260808 01:49:16.366070 2727 raft_consensus.cc:2468] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.369213 2819 ts_tablet_manager.cc:1434] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d: Time spent starting tablet: real 0.134s user 0.006s sys 0.005s
I20260808 01:49:16.375100 2033 catalog_manager.cc:5697] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca reported cstate change: term changed from 0 to 1, leader changed from <none> to 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65). New cstate: current_term: 1 leader_uuid: "085f84d6c0e44b1995bafb3d7a969cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:16.376600 2033 catalog_manager.cc:5697] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca reported cstate change: term changed from 0 to 1, leader changed from <none> to 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65). New cstate: current_term: 1 leader_uuid: "085f84d6c0e44b1995bafb3d7a969cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
W20260808 01:49:16.393507 2774 tablet.cc:2406] T a0cc3757fca846a189aae3a943265a08 P e910e7a0a64b46568c6c3c97e9fc412d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:16.449230 2824 raft_consensus.cc:493] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.449755 2824 raft_consensus.cc:515] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.450457 2826 raft_consensus.cc:493] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.450842 2826 raft_consensus.cc:515] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.451637 2824 leader_election.cc:290] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539), 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873)
I20260808 01:49:16.452807 2826 leader_election.cc:290] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539), 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
I20260808 01:49:16.452777 2727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.452886 2272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085f84d6c0e44b1995bafb3d7a969cca" is_pre_election: true
I20260808 01:49:16.453413 2727 raft_consensus.cc:2468] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 0.
I20260808 01:49:16.453495 2272 raft_consensus.cc:2468] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 0.
I20260808 01:49:16.453603 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:16.454224 2724 raft_consensus.cc:2468] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.454479 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:16.454710 2432 leader_election.cc:304] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6; no voters:
I20260808 01:49:16.455098 2499 raft_consensus.cc:2468] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 085f84d6c0e44b1995bafb3d7a969cca in term 0.
I20260808 01:49:16.455559 2824 raft_consensus.cc:2804] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.455523 2206 leader_election.cc:304] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca, e910e7a0a64b46568c6c3c97e9fc412d; no voters:
I20260808 01:49:16.455932 2824 raft_consensus.cc:493] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.456296 2824 raft_consensus.cc:3060] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.456310 2826 raft_consensus.cc:2804] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:16.456637 2826 raft_consensus.cc:493] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:16.456970 2826 raft_consensus.cc:3060] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.463491 2824 raft_consensus.cc:515] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.463491 2826 raft_consensus.cc:515] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.465543 2824 leader_election.cc:290] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 election: Requested vote from peers e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539), 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873)
I20260808 01:49:16.466154 2826 leader_election.cc:290] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Requested vote from peers e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539), 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
I20260808 01:49:16.466701 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6"
I20260808 01:49:16.466611 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.466711 2272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085f84d6c0e44b1995bafb3d7a969cca"
I20260808 01:49:16.467239 2724 raft_consensus.cc:3060] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.467525 2499 raft_consensus.cc:2393] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 085f84d6c0e44b1995bafb3d7a969cca in current term 1: Already voted for candidate 4e46a7579cc348569736e0e431dbddb6 in this term.
I20260808 01:49:16.467545 2272 raft_consensus.cc:2393] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e46a7579cc348569736e0e431dbddb6 in current term 1: Already voted for candidate 085f84d6c0e44b1995bafb3d7a969cca in this term.
I20260808 01:49:16.467864 2727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b88d1c1afe054fd3b0557b47c5238cf8" candidate_uuid: "085f84d6c0e44b1995bafb3d7a969cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:16.469440 2206 leader_election.cc:304] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 085f84d6c0e44b1995bafb3d7a969cca; no voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d
I20260808 01:49:16.470270 2826 raft_consensus.cc:2749] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:16.473088 2724 raft_consensus.cc:2468] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:16.474205 2433 leader_election.cc:304] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:16.474937 2824 raft_consensus.cc:2804] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:16.475392 2824 raft_consensus.cc:697] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Becoming Leader. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Running, Role: LEADER
I20260808 01:49:16.476171 2824 consensus_queue.cc:237] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:16.485074 2032 catalog_manager.cc:5697] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:16.609756 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 2093
I20260808 01:49:16.662015 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:37873
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:41533
--webserver_port=39357
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20260808 01:49:16.722898 2847 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.723383 2847 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:16.726248 2847 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
W20260808 01:49:16.729419 2661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
W20260808 01:49:16.735270 2661 leader_election.cc:336] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:16.739537 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:16.740357 2499 raft_consensus.cc:2393] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e910e7a0a64b46568c6c3c97e9fc412d in current term 1: Already voted for candidate 085f84d6c0e44b1995bafb3d7a969cca in this term.
I20260808 01:49:16.741904 2663 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6
I20260808 01:49:16.742864 2847 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.747637 2847 raft_consensus.cc:2749] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:49:16.776419 2432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
W20260808 01:49:16.780671 2432 consensus_peers.cc:597] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:16.887928 2824 consensus_queue.cc:1048] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:49:17.011359 2432 consensus_peers.cc:597] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:17.016187 2858 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:17.016760 2858 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:17.017400 2858 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:17.052089 2858 flags.cc:432] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260808 01:49:17.052575 2858 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:17.053520 2858 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:49:17.091826 2858 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:37873
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=39357
--tserver_master_addrs=127.29.180.126:41533
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:17.093690 2858 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:49:17.094662 2858 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:49:17.094854 2858 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:49:17.095660 2824 consensus_queue.cc:1048] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:17.096083 2858 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:17.115371 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:17.994441 2824 raft_consensus.cc:493] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:17.995474 2824 raft_consensus.cc:515] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.000772 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:18.001791 2724 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:18.004467 2433 leader_election.cc:304] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters:
I20260808 01:49:18.006049 2824 leader_election.cc:290] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:18.007668 2824 raft_consensus.cc:2804] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:18.009057 2824 raft_consensus.cc:493] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:18.010162 2824 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:18.027498 2432 leader_election.cc:336] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.039304 2824 raft_consensus.cc:515] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
W20260808 01:49:18.050493 2432 leader_election.cc:336] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.054240 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa85170369884692b5cf1abe2fc79afd" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:18.055160 2724 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:18.073899 2824 leader_election.cc:290] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Requested vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:18.076527 2724 raft_consensus.cc:2468] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 2.
I20260808 01:49:18.078828 2433 leader_election.cc:304] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:18.080127 2824 raft_consensus.cc:2804] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:18.081133 2824 raft_consensus.cc:697] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [term 2 LEADER]: Becoming Leader. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Running, Role: LEADER
I20260808 01:49:18.083083 2824 consensus_queue.cc:237] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.094990 2032 catalog_manager.cc:5697] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: term changed from 1 to 2, leader changed from 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) to 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66). New cstate: current_term: 2 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:18.120802 2880 raft_consensus.cc:493] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:18.126816 2880 raft_consensus.cc:515] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.138098 2499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46a7579cc348569736e0e431dbddb6" is_pre_election: true
I20260808 01:49:18.139202 2499 raft_consensus.cc:2393] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e910e7a0a64b46568c6c3c97e9fc412d in current term 1: Already voted for candidate 085f84d6c0e44b1995bafb3d7a969cca in this term.
I20260808 01:49:18.145130 2880 leader_election.cc:290] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
W20260808 01:49:18.169394 2661 leader_election.cc:336] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.170003 2661 leader_election.cc:304] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca, 4e46a7579cc348569736e0e431dbddb6
I20260808 01:49:18.170945 2880 raft_consensus.cc:3060] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:18.177321 2880 raft_consensus.cc:2749] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:18.178030 2824 raft_consensus.cc:493] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:18.178500 2824 raft_consensus.cc:515] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.183557 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:18.184564 2724 raft_consensus.cc:2468] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:18.192673 2433 leader_election.cc:304] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters:
I20260808 01:49:18.194197 2878 raft_consensus.cc:2804] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:18.194728 2878 raft_consensus.cc:493] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:18.195150 2878 raft_consensus.cc:3060] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:18.203804 2824 leader_election.cc:290] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
W20260808 01:49:18.208622 2432 leader_election.cc:336] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.215866 2878 raft_consensus.cc:515] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
W20260808 01:49:18.236968 2432 leader_election.cc:336] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.241230 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef96d18afb024e62a17e2c0f0f035c5b" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:18.242149 2724 raft_consensus.cc:3060] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:18.251152 2878 leader_election.cc:290] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Requested vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:18.254609 2724 raft_consensus.cc:2468] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 2.
I20260808 01:49:18.262873 2433 leader_election.cc:304] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:18.264950 2878 raft_consensus.cc:2804] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:18.267256 2878 raft_consensus.cc:697] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 2 LEADER]: Becoming Leader. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Running, Role: LEADER
I20260808 01:49:18.269392 2878 consensus_queue.cc:237] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.294382 2032 catalog_manager.cc:5697] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: term changed from 1 to 2, leader changed from 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) to 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66). New cstate: current_term: 2 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:18.386788 2878 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:18.387601 2878 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
W20260808 01:49:18.400524 2432 leader_election.cc:336] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.404611 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" is_pre_election: true
I20260808 01:49:18.405571 2724 raft_consensus.cc:2468] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 1.
I20260808 01:49:18.409961 2433 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:18.411473 2824 raft_consensus.cc:2804] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:18.411999 2824 raft_consensus.cc:493] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:18.412436 2824 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:18.429539 2878 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
I20260808 01:49:18.445850 2824 raft_consensus.cc:515] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.454073 2724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4c031377aec49afaa707e2cbeb840d2" candidate_uuid: "4e46a7579cc348569736e0e431dbddb6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e910e7a0a64b46568c6c3c97e9fc412d"
I20260808 01:49:18.454947 2724 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:18.473013 2824 leader_election.cc:290] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Requested vote from peers 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873), e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539)
W20260808 01:49:18.476256 2432 leader_election.cc:336] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111)
I20260808 01:49:18.478871 2724 raft_consensus.cc:2468] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e46a7579cc348569736e0e431dbddb6 in term 2.
I20260808 01:49:18.480902 2433 leader_election.cc:304] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e46a7579cc348569736e0e431dbddb6, e910e7a0a64b46568c6c3c97e9fc412d; no voters: 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:18.482699 2824 raft_consensus.cc:2804] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:18.484510 2824 raft_consensus.cc:697] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [term 2 LEADER]: Becoming Leader. State: Replica: 4e46a7579cc348569736e0e431dbddb6, State: Running, Role: LEADER
I20260808 01:49:18.485694 2824 consensus_queue.cc:237] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:18.504289 2032 catalog_manager.cc:5697] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: term changed from 1 to 2, leader changed from 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) to 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66). New cstate: current_term: 2 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
W20260808 01:49:18.516520 2432 consensus_peers.cc:597] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:18.610674 2724 raft_consensus.cc:1275] T fa85170369884692b5cf1abe2fc79afd P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:49:18.612541 2878 consensus_queue.cc:1048] T fa85170369884692b5cf1abe2fc79afd P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:49:18.516691 2871 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 2858
W20260808 01:49:18.770877 2871 kernel_stack_watchdog.cc:198] Thread 2858 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:49:17.117106 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:18.772236 2858 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.656s user 0.605s sys 0.964s
W20260808 01:49:18.772701 2858 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.656s user 0.605s sys 0.964s
W20260808 01:49:18.773857 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:18.776729 2874 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1654 milliseconds
I20260808 01:49:18.776741 2858 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:18.778195 2858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:18.780419 2858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:18.781787 2858 hybrid_clock.cc:648] HybridClock initialized: now 1786153758781748 us; error 53 us; skew 500 ppm
I20260808 01:49:18.782719 2858 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:18.789439 2858 webserver.cc:533] Webserver started at http://127.29.180.65:39357/ using document root <none> and password file <none>
I20260808 01:49:18.791330 2858 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:18.791623 2858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:49:18.795047 2858 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:49:18.800144 2858 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:49:18.816078 2858 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:49:18.872740 2727 raft_consensus.cc:1275] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:49:18.874369 2893 consensus_queue.cc:1048] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:49:18.881332 2432 consensus_peers.cc:597] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:18.886291 2724 raft_consensus.cc:1275] T d4c031377aec49afaa707e2cbeb840d2 P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:49:18.888247 2824 consensus_queue.cc:1048] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:49:18.905889 2432 consensus_peers.cc:597] T d4c031377aec49afaa707e2cbeb840d2 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:18.961748 2858 dir_manager.cc:319] Skip initializing rocksdb instance for the non-healthy directory /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data
I20260808 01:49:19.287142 2858 fs_manager.cc:714] Time spent opening directory manager: real 0.489s user 0.227s sys 0.219s
E20260808 01:49:19.291762 2858 log_block_manager.cc:3241] Not using report from /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260808 01:49:19.294348 2858 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 01:49:19.294696 2858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "085f84d6c0e44b1995bafb3d7a969cca"
format_stamp: "Formatted at 2026-08-08 01:49:09 on dist-test-slave-p3t5"
I20260808 01:49:19.295235 2858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:19.341735 2858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:19.343119 2858 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:19.343612 2858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:19.345710 2858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:19.350945 2981 ts_tablet_manager.cc:548] Loading tablet metadata (0/5 complete)
W20260808 01:49:19.364800 2984 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet ef96d18afb024e62a17e2c0f0f035c5b: could not find data dir with uuid cbbfa0b1a83a42b58553ca3af2393a0c
W20260808 01:49:19.369379 2985 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet b88d1c1afe054fd3b0557b47c5238cf8: could not find data dir with uuid cbbfa0b1a83a42b58553ca3af2393a0c
I20260808 01:49:19.371562 2858 ts_tablet_manager.cc:585] Loaded tablet metadata (5 total tablets, 5 live tablets)
I20260808 01:49:19.371831 2858 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.022s user 0.011s sys 0.007s
I20260808 01:49:19.372145 2858 ts_tablet_manager.cc:600] Registering tablets (0/5 complete)
I20260808 01:49:19.378268 2983 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:19.381465 2985 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:19.384878 2986 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:19.392441 2984 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
E20260808 01:49:19.393220 2984 ts_tablet_manager.cc:1425] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:19.393759 2984 ts_tablet_manager.cc:1403] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 01:49:19.394280 2984 tablet_replica.cc:333] stopping tablet replica
E20260808 01:49:19.396498 2986 ts_tablet_manager.cc:1425] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef96d18afb024e62a17e2c0f0f035c5b
I20260808 01:49:19.396894 2858 ts_tablet_manager.cc:616] Registered 5 tablets
I20260808 01:49:19.397171 2986 ts_tablet_manager.cc:1403] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260808 01:49:19.397596 2986 tablet_replica.cc:333] stopping tablet replica
I20260808 01:49:19.397229 2858 ts_tablet_manager.cc:595] Time spent register tablets: real 0.025s user 0.025s sys 0.001s
I20260808 01:49:19.398020 2981 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:19.398025 2986 raft_consensus.cc:2243] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:19.416333 2986 raft_consensus.cc:2272] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 01:49:19.421034 2432 consensus_peers.cc:597] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:49:19.394815 2984 raft_consensus.cc:2243] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:19.425391 2984 raft_consensus.cc:2272] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:19.449501 2983 log.cc:826] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:19.477934 2983 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:19.479678 2983 tablet_bootstrap.cc:492] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap complete.
I20260808 01:49:19.480983 2985 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 01:49:19.482439 2985 tablet_bootstrap.cc:492] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap complete.
I20260808 01:49:19.484349 2985 ts_tablet_manager.cc:1403] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.103s user 0.045s sys 0.042s
I20260808 01:49:19.486624 2983 ts_tablet_manager.cc:1403] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.109s user 0.040s sys 0.045s
W20260808 01:49:19.506103 2432 consensus_peers.cc:597] T a0cc3757fca846a189aae3a943265a08 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:37873: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:49:19.507642 2981 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 01:49:19.509179 2981 tablet_bootstrap.cc:492] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap complete.
I20260808 01:49:19.511121 2981 ts_tablet_manager.cc:1403] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.113s user 0.047s sys 0.043s
I20260808 01:49:19.523761 2981 raft_consensus.cc:359] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.525040 2981 raft_consensus.cc:740] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:19.526189 2983 raft_consensus.cc:359] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.526113 2981 consensus_queue.cc:260] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.527005 2983 raft_consensus.cc:740] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:19.528007 2985 raft_consensus.cc:359] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.531311 2985 raft_consensus.cc:740] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:19.532231 2985 consensus_queue.cc:260] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.529892 2983 consensus_queue.cc:260] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:19.548579 2985 ts_tablet_manager.cc:1434] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.064s user 0.032s sys 0.010s
I20260808 01:49:19.549292 2981 ts_tablet_manager.cc:1434] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.038s user 0.006s sys 0.022s
I20260808 01:49:19.550244 2983 ts_tablet_manager.cc:1434] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.063s user 0.017s sys 0.001s
I20260808 01:49:19.689268 2858 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:37873
I20260808 01:49:19.689582 3101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:37873 every 8 connection(s)
I20260808 01:49:19.692058 2858 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:49:19.697418 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 2858
W20260808 01:49:19.697021 3103 tablet.cc:2406] T a0cc3757fca846a189aae3a943265a08 P 085f84d6c0e44b1995bafb3d7a969cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:19.728608 3102 heartbeater.cc:344] Connected to a master server at 127.29.180.126:41533
I20260808 01:49:19.729163 3102 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:19.730618 3102 heartbeater.cc:507] Master 127.29.180.126:41533 requested a full tablet report, sending...
I20260808 01:49:19.736073 2032 ts_manager.cc:194] Re-registered known tserver with Master: 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873)
W20260808 01:49:19.738752 2032 catalog_manager.cc:5610] Tablet ef96d18afb024e62a17e2c0f0f035c5b (table test-workload [id=4e15842e93314742887b2318731f2757]) has failed on TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef96d18afb024e62a17e2c0f0f035c5b
W20260808 01:49:19.739300 2032 catalog_manager.cc:5610] Tablet b88d1c1afe054fd3b0557b47c5238cf8 (table test-workload [id=4e15842e93314742887b2318731f2757]) has failed on TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:19.741992 2032 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:46813
I20260808 01:49:19.759428 3057 raft_consensus.cc:3060] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:19.805998 3056 raft_consensus.cc:3060] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:19.811322 3056 pending_rounds.cc:85] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Aborting all ops after (but not including) 0
I20260808 01:49:19.811643 3056 pending_rounds.cc:107] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:49:19.811959 3056 raft_consensus.cc:2889] T fa85170369884692b5cf1abe2fc79afd P 085f84d6c0e44b1995bafb3d7a969cca [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:49:19.812737 3057 pending_rounds.cc:85] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Aborting all ops after (but not including) 0
I20260808 01:49:19.813109 3102 heartbeater.cc:499] Master 127.29.180.126:41533 was elected leader, sending a full tablet report...
I20260808 01:49:19.813122 3057 pending_rounds.cc:107] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:49:19.813539 3057 raft_consensus.cc:2889] T d4c031377aec49afaa707e2cbeb840d2 P 085f84d6c0e44b1995bafb3d7a969cca [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:49:19.829351 2032 catalog_manager.cc:5610] Tablet ef96d18afb024e62a17e2c0f0f035c5b (table test-workload [id=4e15842e93314742887b2318731f2757]) has failed on TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet ef96d18afb024e62a17e2c0f0f035c5b
W20260808 01:49:19.830029 2032 catalog_manager.cc:5610] Tablet b88d1c1afe054fd3b0557b47c5238cf8 (table test-workload [id=4e15842e93314742887b2318731f2757]) has failed on TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:19.889698 2824 raft_consensus.cc:1012] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Attempting to remove follower 085f84d6c0e44b1995bafb3d7a969cca from the Raft config. Reason: The tablet replica hosted on peer 085f84d6c0e44b1995bafb3d7a969cca has failed
I20260808 01:49:19.891708 2824 consensus_queue.cc:237] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:19.896559 2724 raft_consensus.cc:1275] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:19.897904 2994 consensus_queue.cc:1048] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:19.905084 2824 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:19.906850 2724 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:19.917039 2033 catalog_manager.cc:5697] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:19.935734 2499 consensus_queue.cc:237] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
W20260808 01:49:19.941744 2432 consensus_peers.cc:597] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet b88d1c1afe054fd3b0557b47c5238cf8. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:19.942471 2724 raft_consensus.cc:1275] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:19.943791 2994 consensus_queue.cc:1048] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:19.947875 3037 tablet_service.cc:1558] Processing DeleteTablet for tablet b88d1c1afe054fd3b0557b47c5238cf8 with delete_type TABLET_DATA_TOMBSTONED (TS 085f84d6c0e44b1995bafb3d7a969cca not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:53192
I20260808 01:49:19.949879 3127 ts_tablet_manager.cc:1916] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:19.950590 2824 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:19.952283 2724 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:19.961544 2020 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet b88d1c1afe054fd3b0557b47c5238cf8 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:49:19.963922 2033 catalog_manager.cc:5697] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d reported cstate change: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:19.981704 3127 ts_tablet_manager.cc:1929] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:19.982169 3127 log.cc:1199] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:19.984315 2018 catalog_manager.cc:5028] TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): tablet b88d1c1afe054fd3b0557b47c5238cf8 (table test-workload [id=4e15842e93314742887b2318731f2757]) successfully deleted
I20260808 01:49:20.052287 2994 raft_consensus.cc:1012] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Attempting to remove follower 085f84d6c0e44b1995bafb3d7a969cca from the Raft config. Reason: The tablet replica hosted on peer 085f84d6c0e44b1995bafb3d7a969cca has failed
I20260808 01:49:20.054001 2994 consensus_queue.cc:237] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } }
I20260808 01:49:20.059005 2724 raft_consensus.cc:1275] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:20.060370 2824 consensus_queue.cc:1048] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:20.067400 2994 raft_consensus.cc:2955] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:20.069077 2724 raft_consensus.cc:2955] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:20.077041 2033 catalog_manager.cc:5697] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: config changed from index -1 to 2, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } }
I20260808 01:49:20.086143 3037 tablet_service.cc:1558] Processing DeleteTablet for tablet ef96d18afb024e62a17e2c0f0f035c5b with delete_type TABLET_DATA_TOMBSTONED (TS 085f84d6c0e44b1995bafb3d7a969cca not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:53192
I20260808 01:49:20.088358 3127 ts_tablet_manager.cc:1916] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:20.088904 2499 consensus_queue.cc:237] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
W20260808 01:49:20.096311 2432 consensus_peers.cc:597] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:20.097784 2724 raft_consensus.cc:1275] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:20.099522 2824 consensus_queue.cc:1048] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.104632 3127 ts_tablet_manager.cc:1929] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:20.105062 3127 log.cc:1199] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/ef96d18afb024e62a17e2c0f0f035c5b
I20260808 01:49:20.107131 2018 catalog_manager.cc:5028] TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): tablet ef96d18afb024e62a17e2c0f0f035c5b (table test-workload [id=4e15842e93314742887b2318731f2757]) successfully deleted
I20260808 01:49:20.109920 3128 raft_consensus.cc:2955] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:20.115638 2727 raft_consensus.cc:2955] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:20.140407 2020 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet ef96d18afb024e62a17e2c0f0f035c5b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:49:20.141965 2032 catalog_manager.cc:5697] T ef96d18afb024e62a17e2c0f0f035c5b P e910e7a0a64b46568c6c3c97e9fc412d reported cstate change: config changed from index 2 to 3, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New cstate: current_term: 2 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:20.163182 3037 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:20.194783 2478 tablet_service.cc:1467] Tablet server has 5 leaders and 0 scanners
I20260808 01:49:20.200320 2706 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:20.338893 2824 raft_consensus.cc:1012] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Attempting to remove follower 085f84d6c0e44b1995bafb3d7a969cca from the Raft config. Reason: The tablet replica hosted on peer 085f84d6c0e44b1995bafb3d7a969cca has failed
I20260808 01:49:20.340628 2824 consensus_queue.cc:237] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } }
I20260808 01:49:20.345714 2727 raft_consensus.cc:1275] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:20.347193 2824 consensus_queue.cc:1048] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:20.353399 3128 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:20.360916 2727 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:20.364781 2031 catalog_manager.cc:5697] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: config changed from index 3 to 4, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } }
I20260808 01:49:20.378051 3037 tablet_service.cc:1558] Processing DeleteTablet for tablet b88d1c1afe054fd3b0557b47c5238cf8 with delete_type TABLET_DATA_TOMBSTONED (TS 085f84d6c0e44b1995bafb3d7a969cca not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53192
I20260808 01:49:20.380545 2018 catalog_manager.cc:5028] TS 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): tablet b88d1c1afe054fd3b0557b47c5238cf8 (table test-workload [id=4e15842e93314742887b2318731f2757]) successfully deleted
I20260808 01:49:20.381186 2499 consensus_queue.cc:237] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
W20260808 01:49:20.387220 2432 consensus_peers.cc:597] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 -> Peer 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): Couldn't send request to peer 085f84d6c0e44b1995bafb3d7a969cca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:20.387646 2727 raft_consensus.cc:1275] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Refusing update from remote peer 4e46a7579cc348569736e0e431dbddb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 01:49:20.389019 2824 consensus_queue.cc:1048] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:20.394747 3128 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:20.396556 2727 raft_consensus.cc:2955] T b88d1c1afe054fd3b0557b47c5238cf8 P e910e7a0a64b46568c6c3c97e9fc412d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
I20260808 01:49:20.405920 2020 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:VOTER RPC for tablet b88d1c1afe054fd3b0557b47c5238cf8 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260808 01:49:20.409117 2032 catalog_manager.cc:5697] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6 reported cstate change: config changed from index 4 to 5, VOTER 085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65) added. New cstate: current_term: 1 leader_uuid: "4e46a7579cc348569736e0e431dbddb6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
079165bb7d084b7d9222554b096fbd3e | 127.29.180.126:41533 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
085f84d6c0e44b1995bafb3d7a969cca | 127.29.180.65:37873 | HEALTHY | <none> | 0 | 0
4e46a7579cc348569736e0e431dbddb6 | 127.29.180.66:42617 | HEALTHY | <none> | 5 | 0
e910e7a0a64b46568c6c3c97e9fc412d | 127.29.180.67:42539 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:37873
env_inject_eio | 1 | hidden | 127.29.180.65:37873
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:37873
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:37873
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:42617
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:42539
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:37873
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:42617
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:42539
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet b88d1c1afe054fd3b0557b47c5238cf8 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): RUNNING
4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617): RUNNING [LEADER]
085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = e910e7a0a64b46568c6c3c97e9fc412d
B = 4e46a7579cc348569736e0e431dbddb6
C = 085f84d6c0e44b1995bafb3d7a969cca
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
Tablet ef96d18afb024e62a17e2c0f0f035c5b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617): RUNNING [LEADER]
e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): RUNNING
All reported replicas are:
A = 085f84d6c0e44b1995bafb3d7a969cca
B = 4e46a7579cc348569736e0e431dbddb6
C = e910e7a0a64b46568c6c3c97e9fc412d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 3 | Yes
C | A B* C | 2 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 5 | 3 | 0 | 2 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:49:20.472765 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:49:20.522132 3150 ts_tablet_manager.cc:933] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Initiating tablet copy from peer 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
I20260808 01:49:20.523430 3150 tablet_copy_client.cc:286] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:49:20.523993 3150 tablet_copy_client.cc:323] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:42617
I20260808 01:49:20.534150 2520 tablet_copy_service.cc:140] P 4e46a7579cc348569736e0e431dbddb6: Received BeginTabletCopySession request for tablet ef96d18afb024e62a17e2c0f0f035c5b from peer 085f84d6c0e44b1995bafb3d7a969cca ({username='slave'} at 127.29.180.65:54497)
I20260808 01:49:20.534653 2520 tablet_copy_service.cc:161] P 4e46a7579cc348569736e0e431dbddb6: Beginning new tablet copy session on tablet ef96d18afb024e62a17e2c0f0f035c5b from peer 085f84d6c0e44b1995bafb3d7a969cca at {username='slave'} at 127.29.180.65:54497: session id = 085f84d6c0e44b1995bafb3d7a969cca-ef96d18afb024e62a17e2c0f0f035c5b
I20260808 01:49:20.539882 2520 tablet_copy_source_session.cc:215] T ef96d18afb024e62a17e2c0f0f035c5b P 4e46a7579cc348569736e0e431dbddb6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:20.542824 3150 ts_tablet_manager.cc:1916] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:49:20.554095 3150 ts_tablet_manager.cc:1929] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:49:20.563330 3150 tablet_copy_client.cc:806] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:20.564077 3150 tablet_copy_client.cc:670] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:20.568902 3150 tablet_copy_client.cc:538] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:20.575722 3150 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:20.595713 3150 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:20.596664 3150 tablet_bootstrap.cc:492] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap complete.
I20260808 01:49:20.597252 3150 ts_tablet_manager.cc:1403] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.009s
I20260808 01:49:20.599390 3150 raft_consensus.cc:359] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
I20260808 01:49:20.599918 3150 raft_consensus.cc:740] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:20.600531 3150 consensus_queue.cc:260] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
I20260808 01:49:20.602283 3150 ts_tablet_manager.cc:1434] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20260808 01:49:20.604115 2520 tablet_copy_service.cc:342] P 4e46a7579cc348569736e0e431dbddb6: Request end of tablet copy session 085f84d6c0e44b1995bafb3d7a969cca-ef96d18afb024e62a17e2c0f0f035c5b received from {username='slave'} at 127.29.180.65:54497
I20260808 01:49:20.604521 2520 tablet_copy_service.cc:434] P 4e46a7579cc348569736e0e431dbddb6: ending tablet copy session 085f84d6c0e44b1995bafb3d7a969cca-ef96d18afb024e62a17e2c0f0f035c5b on tablet ef96d18afb024e62a17e2c0f0f035c5b with peer 085f84d6c0e44b1995bafb3d7a969cca
E20260808 01:49:20.724288 3098 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:49:20.762426 3150 ts_tablet_manager.cc:933] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Initiating tablet copy from peer 4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617)
I20260808 01:49:20.763928 3150 tablet_copy_client.cc:286] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid
I20260808 01:49:20.764515 3150 tablet_copy_client.cc:323] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:42617
I20260808 01:49:20.766373 2520 tablet_copy_service.cc:140] P 4e46a7579cc348569736e0e431dbddb6: Received BeginTabletCopySession request for tablet b88d1c1afe054fd3b0557b47c5238cf8 from peer 085f84d6c0e44b1995bafb3d7a969cca ({username='slave'} at 127.29.180.65:54497)
I20260808 01:49:20.766901 2520 tablet_copy_service.cc:161] P 4e46a7579cc348569736e0e431dbddb6: Beginning new tablet copy session on tablet b88d1c1afe054fd3b0557b47c5238cf8 from peer 085f84d6c0e44b1995bafb3d7a969cca at {username='slave'} at 127.29.180.65:54497: session id = 085f84d6c0e44b1995bafb3d7a969cca-b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:20.774030 2520 tablet_copy_source_session.cc:215] T b88d1c1afe054fd3b0557b47c5238cf8 P 4e46a7579cc348569736e0e431dbddb6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:20.776664 3150 ts_tablet_manager.cc:1916] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:49:20.787987 3150 ts_tablet_manager.cc:1929] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown
I20260808 01:49:20.794090 3150 tablet_copy_client.cc:806] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:20.794705 3150 tablet_copy_client.cc:670] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:20.798980 3150 tablet_copy_client.cc:538] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:20.805401 3150 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap starting.
I20260808 01:49:20.836191 3150 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:20.837404 3150 tablet_bootstrap.cc:492] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Bootstrap complete.
I20260808 01:49:20.838080 3150 ts_tablet_manager.cc:1403] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.005s
I20260808 01:49:20.841977 3150 raft_consensus.cc:359] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
I20260808 01:49:20.842454 3150 raft_consensus.cc:740] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085f84d6c0e44b1995bafb3d7a969cca, State: Initialized, Role: FOLLOWER
I20260808 01:49:20.843050 3150 consensus_queue.cc:260] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e910e7a0a64b46568c6c3c97e9fc412d" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 42539 } } peers { permanent_uuid: "4e46a7579cc348569736e0e431dbddb6" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 42617 } } peers { permanent_uuid: "085f84d6c0e44b1995bafb3d7a969cca" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 37873 } }
I20260808 01:49:20.844954 3150 ts_tablet_manager.cc:1434] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260808 01:49:20.867339 2520 tablet_copy_service.cc:342] P 4e46a7579cc348569736e0e431dbddb6: Request end of tablet copy session 085f84d6c0e44b1995bafb3d7a969cca-b88d1c1afe054fd3b0557b47c5238cf8 received from {username='slave'} at 127.29.180.65:54497
I20260808 01:49:20.868259 2520 tablet_copy_service.cc:434] P 4e46a7579cc348569736e0e431dbddb6: ending tablet copy session 085f84d6c0e44b1995bafb3d7a969cca-b88d1c1afe054fd3b0557b47c5238cf8 on tablet b88d1c1afe054fd3b0557b47c5238cf8 with peer 085f84d6c0e44b1995bafb3d7a969cca
I20260808 01:49:20.949301 3037 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:20.951946 2478 tablet_service.cc:1467] Tablet server has 5 leaders and 0 scanners
I20260808 01:49:20.952688 2706 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:20.965878 3056 raft_consensus.cc:1217] T ef96d18afb024e62a17e2c0f0f035c5b P 085f84d6c0e44b1995bafb3d7a969cca [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
079165bb7d084b7d9222554b096fbd3e | 127.29.180.126:41533 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
085f84d6c0e44b1995bafb3d7a969cca | 127.29.180.65:37873 | HEALTHY | <none> | 0 | 0
4e46a7579cc348569736e0e431dbddb6 | 127.29.180.66:42617 | HEALTHY | <none> | 5 | 0
e910e7a0a64b46568c6c3c97e9fc412d | 127.29.180.67:42539 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:37873
env_inject_eio | 1 | hidden | 127.29.180.65:37873
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:37873
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:37873
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:42617
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:42539
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:37873
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:42617
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:42539
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet b88d1c1afe054fd3b0557b47c5238cf8 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): RUNNING
4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617): RUNNING [LEADER]
085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): RUNNING
All reported replicas are:
A = e910e7a0a64b46568c6c3c97e9fc412d
B = 4e46a7579cc348569736e0e431dbddb6
C = 085f84d6c0e44b1995bafb3d7a969cca
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 5 | Yes
B | A B* C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
Tablet ef96d18afb024e62a17e2c0f0f035c5b of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
4e46a7579cc348569736e0e431dbddb6 (127.29.180.66:42617): RUNNING [LEADER]
e910e7a0a64b46568c6c3c97e9fc412d (127.29.180.67:42539): RUNNING
085f84d6c0e44b1995bafb3d7a969cca (127.29.180.65:37873): RUNNING
All reported replicas are:
A = 4e46a7579cc348569736e0e431dbddb6
B = e910e7a0a64b46568c6c3c97e9fc412d
C = 085f84d6c0e44b1995bafb3d7a969cca
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 3 | Yes
B | A* B C | 2 | 3 | Yes
C | A B C | 2 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 0 | 2
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:49:21.224294 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:49:21.342923 3056 raft_consensus.cc:1217] T b88d1c1afe054fd3b0557b47c5238cf8 P 085f84d6c0e44b1995bafb3d7a969cca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260808 01:49:21.755383 3037 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:21.781266 2478 tablet_service.cc:1467] Tablet server has 5 leaders and 0 scanners
I20260808 01:49:21.786791 2706 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
079165bb7d084b7d9222554b096fbd3e | 127.29.180.126:41533 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
085f84d6c0e44b1995bafb3d7a969cca | 127.29.180.65:37873 | HEALTHY | <none> | 0 | 0
4e46a7579cc348569736e0e431dbddb6 | 127.29.180.66:42617 | HEALTHY | <none> | 5 | 0
e910e7a0a64b46568c6c3c97e9fc412d | 127.29.180.67:42539 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:37873
env_inject_eio | 1 | hidden | 127.29.180.65:37873
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:37873
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:37873
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:42617
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:42539
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:37873
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:42617
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_4.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:42539
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:44627 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 5
First Quartile | 5
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:49:22.039702 30417 log_verifier.cc:126] Checking tablet a0cc3757fca846a189aae3a943265a08
I20260808 01:49:22.101404 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a0cc3757fca846a189aae3a943265a08
I20260808 01:49:22.101687 30417 log_verifier.cc:126] Checking tablet b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:22.167881 30417 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b88d1c1afe054fd3b0557b47c5238cf8
I20260808 01:49:22.168102 30417 log_verifier.cc:126] Checking tablet d4c031377aec49afaa707e2cbeb840d2
I20260808 01:49:22.248558 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d4c031377aec49afaa707e2cbeb840d2
I20260808 01:49:22.248873 30417 log_verifier.cc:126] Checking tablet ef96d18afb024e62a17e2c0f0f035c5b
I20260808 01:49:22.323580 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ef96d18afb024e62a17e2c0f0f035c5b
I20260808 01:49:22.323845 30417 log_verifier.cc:126] Checking tablet fa85170369884692b5cf1abe2fc79afd
I20260808 01:49:22.388016 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fa85170369884692b5cf1abe2fc79afd
I20260808 01:49:22.941177 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 2858
I20260808 01:49:23.009212 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 2322
I20260808 01:49:23.068732 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 2549
I20260808 01:49:23.123117 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 1909
2026-08-08T01:49:23Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/4 (18249 ms)
[ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/5
2026-08-08T01:49:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:49:23Z Disabled control of system clock
I20260808 01:49:23.268049 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:40305
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:40305
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:49:23.595657 3228 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:23.596340 3228 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:23.596854 3228 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:23.631086 3228 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:49:23.631489 3228 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:23.631778 3228 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:49:23.632053 3228 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:49:23.676482 3228 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:40305
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:23.678061 3228 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:23.679816 3228 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:23.691550 3234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:25.095789 3233 debug-util.cc:398] Leaking SignalData structure 0x7b08000290e0 after lost signal to thread 3228
W20260808 01:49:25.562400 3233 kernel_stack_watchdog.cc:198] Thread 3228 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:49:23.692143 3235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:25.563619 3228 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.872s user 0.615s sys 1.257s
W20260808 01:49:25.564081 3228 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.872s user 0.615s sys 1.257s
W20260808 01:49:25.566092 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:25.570374 3236 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1877 milliseconds
I20260808 01:49:25.570428 3228 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:25.571858 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:25.574529 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:25.575919 3228 hybrid_clock.cc:648] HybridClock initialized: now 1786153765575896 us; error 96 us; skew 500 ppm
I20260808 01:49:25.576877 3228 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:25.583833 3228 webserver.cc:533] Webserver started at http://127.29.180.126:38093/ using document root <none> and password file <none>
I20260808 01:49:25.585886 3228 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:25.586122 3228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:25.586774 3228 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:25.591567 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:25.592741 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:25.593669 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:25.594547 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-3/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:25.595428 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-4/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:25.596282 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:26.189608 3228 fs_manager.cc:696] Time spent creating directory manager: real 0.593s user 0.237s sys 0.275s
I20260808 01:49:26.198930 3228 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 01:49:26.199334 3228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "3a98afc236b54bde99ac36162e176b62"
format_stamp: "Formatted at 2026-08-08 01:49:25 on dist-test-slave-p3t5"
I20260808 01:49:26.199728 3228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:26.252326 3228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:26.253891 3228 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:26.254416 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:26.326005 3228 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:40305
I20260808 01:49:26.326120 3387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:40305 every 8 connection(s)
I20260808 01:49:26.328712 3228 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:49:26.336522 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 3228
I20260808 01:49:26.337124 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:49:26.361248 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62: Bootstrap starting.
I20260808 01:49:26.368772 3388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:26.370935 3388 log.cc:826] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:26.375784 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62: No bootstrap required, opened a new log
I20260808 01:49:26.395325 3388 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } }
I20260808 01:49:26.396114 3388 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:26.396391 3388 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a98afc236b54bde99ac36162e176b62, State: Initialized, Role: FOLLOWER
I20260808 01:49:26.397181 3388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } }
I20260808 01:49:26.397778 3388 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:49:26.398087 3388 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:49:26.398444 3388 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:26.403426 3388 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } }
I20260808 01:49:26.404263 3388 leader_election.cc:304] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a98afc236b54bde99ac36162e176b62; no voters:
I20260808 01:49:26.406150 3388 leader_election.cc:290] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:49:26.406742 3394 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:26.409405 3394 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [term 1 LEADER]: Becoming Leader. State: Replica: 3a98afc236b54bde99ac36162e176b62, State: Running, Role: LEADER
I20260808 01:49:26.410529 3394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } }
I20260808 01:49:26.411851 3388 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:49:26.421134 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a98afc236b54bde99ac36162e176b62. Latest consensus state: current_term: 1 leader_uuid: "3a98afc236b54bde99ac36162e176b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } } }
I20260808 01:49:26.422117 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:26.422163 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a98afc236b54bde99ac36162e176b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a98afc236b54bde99ac36162e176b62" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 40305 } } }
I20260808 01:49:26.422847 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62 [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:26.425482 3402 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:49:26.437415 3402 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:49:26.456125 3402 catalog_manager.cc:1383] Generated new cluster ID: 5612b2cd0e754145837cf64d4f442a56
I20260808 01:49:26.456442 3402 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:49:26.483593 3402 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:49:26.485143 3402 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:49:26.505198 3402 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 3a98afc236b54bde99ac36162e176b62: Generated new TSK 0
I20260808 01:49:26.506209 3402 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:49:26.518467 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:49:26.858572 3413 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:26.859184 3413 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:26.859705 3413 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:26.896128 3413 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:26.897038 3413 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:49:26.935653 3413 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:26.937212 3413 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:26.939023 3413 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:26.953086 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:28.356528 3419 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 3413
W20260808 01:49:28.532655 3413 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.579s user 0.592s sys 0.985s
W20260808 01:49:28.533064 3413 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.579s user 0.592s sys 0.985s
W20260808 01:49:26.955404 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:28.533931 3422 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260808 01:49:28.535151 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:28.535571 3413 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:28.539867 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:28.542598 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:28.544153 3413 hybrid_clock.cc:648] HybridClock initialized: now 1786153768544092 us; error 48 us; skew 500 ppm
I20260808 01:49:28.545305 3413 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:28.553468 3413 webserver.cc:533] Webserver started at http://127.29.180.65:36595/ using document root <none> and password file <none>
I20260808 01:49:28.555733 3413 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:28.556068 3413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:28.556986 3413 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:28.563632 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:28.565287 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:28.566635 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:28.567937 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:28.569180 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:28.570482 3413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:29.185141 3413 fs_manager.cc:696] Time spent creating directory manager: real 0.614s user 0.294s sys 0.234s
I20260808 01:49:29.194413 3413 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260808 01:49:29.194731 3413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:29.195072 3413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:29.279827 3413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:29.281190 3413 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:29.281623 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:29.283677 3413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:29.287364 3413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:29.287560 3413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:29.287744 3413 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:29.287914 3413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:29.427649 3413 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:44729
I20260808 01:49:29.427775 3636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:44729 every 8 connection(s)
I20260808 01:49:29.430267 3413 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:49:29.438568 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 3413
I20260808 01:49:29.439227 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:49:29.446507 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:49:29.478684 3637 heartbeater.cc:344] Connected to a master server at 127.29.180.126:40305
I20260808 01:49:29.479188 3637 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:29.480371 3637 heartbeater.cc:507] Master 127.29.180.126:40305 requested a full tablet report, sending...
I20260808 01:49:29.483372 3352 ts_manager.cc:194] Registered new tserver with Master: 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729)
I20260808 01:49:29.485419 3352 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:41063
W20260808 01:49:29.773064 3641 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:29.773692 3641 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:29.774214 3641 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:29.807343 3641 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:29.808243 3641 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:49:29.846395 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:29.847913 3641 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:29.849676 3641 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:29.863866 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:30.489320 3637 heartbeater.cc:499] Master 127.29.180.126:40305 was elected leader, sending a full tablet report...
W20260808 01:49:29.869462 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:29.871871 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:31.076689 3649 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1207 milliseconds
I20260808 01:49:31.076838 3641 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:31.078240 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:31.081050 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:31.082495 3641 hybrid_clock.cc:648] HybridClock initialized: now 1786153771082440 us; error 72 us; skew 500 ppm
I20260808 01:49:31.083379 3641 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:31.090348 3641 webserver.cc:533] Webserver started at http://127.29.180.66:46297/ using document root <none> and password file <none>
I20260808 01:49:31.091995 3641 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:31.092249 3641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:31.092932 3641 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:31.097422 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.098688 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.099611 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.100502 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-3/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.101374 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-4/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.102376 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.701258 3641 fs_manager.cc:696] Time spent creating directory manager: real 0.598s user 0.282s sys 0.232s
I20260808 01:49:31.710709 3641 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260808 01:49:31.711081 3641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "529e440fab074f228c5eee925f4a3bb5"
format_stamp: "Formatted at 2026-08-08 01:49:31 on dist-test-slave-p3t5"
I20260808 01:49:31.711444 3641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:31.766472 3641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:31.767959 3641 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:31.768455 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:31.770582 3641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:31.774236 3641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:31.774436 3641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:31.774741 3641 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:31.774907 3641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:31.910533 3641 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:33737
I20260808 01:49:31.910667 3863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:33737 every 8 connection(s)
I20260808 01:49:31.913121 3641 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:49:31.922829 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 3641
I20260808 01:49:31.923331 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:49:31.930845 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:49:31.941902 3864 heartbeater.cc:344] Connected to a master server at 127.29.180.126:40305
I20260808 01:49:31.942379 3864 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:31.943473 3864 heartbeater.cc:507] Master 127.29.180.126:40305 requested a full tablet report, sending...
I20260808 01:49:31.945917 3352 ts_manager.cc:194] Registered new tserver with Master: 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:31.947253 3352 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:49967
W20260808 01:49:32.260562 3868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:32.261206 3868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:32.261799 3868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:32.304154 3868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:32.305590 3868 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:49:32.369040 3868 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:32.371217 3868 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:32.373086 3868 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:32.389017 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:32.950873 3864 heartbeater.cc:499] Master 127.29.180.126:40305 was elected leader, sending a full tablet report...
W20260808 01:49:33.793536 3873 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 3868
W20260808 01:49:34.152053 3873 kernel_stack_watchdog.cc:198] Thread 3868 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:49:34.152441 3868 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.762s user 0.652s sys 1.110s
W20260808 01:49:32.390990 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:34.152879 3868 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.763s user 0.652s sys 1.110s
W20260808 01:49:34.153692 3876 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1762 milliseconds
W20260808 01:49:34.154996 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:34.154924 3868 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:34.158582 3868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:34.161023 3868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:34.162446 3868 hybrid_clock.cc:648] HybridClock initialized: now 1786153774162386 us; error 46 us; skew 500 ppm
I20260808 01:49:34.163488 3868 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:34.171123 3868 webserver.cc:533] Webserver started at http://127.29.180.67:40121/ using document root <none> and password file <none>
I20260808 01:49:34.172736 3868 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:34.173019 3868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:34.173834 3868 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:34.178741 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.180063 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.181003 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.182016 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-3/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.183053 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-4/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.183983 3868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.797238 3868 fs_manager.cc:696] Time spent creating directory manager: real 0.613s user 0.311s sys 0.220s
I20260808 01:49:34.807238 3868 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 01:49:34.807622 3868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "cc7f064562454e9586acf4bc50ced2f5"
format_stamp: "Formatted at 2026-08-08 01:49:34 on dist-test-slave-p3t5"
I20260808 01:49:34.808028 3868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:34.862545 3868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:34.863929 3868 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:34.864409 3868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:34.866765 3868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:34.870705 3868 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:34.871018 3868 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:34.871273 3868 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:34.871441 3868 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:35.018306 3868 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:41221
I20260808 01:49:35.018493 4089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:41221 every 8 connection(s)
I20260808 01:49:35.020820 3868 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:49:35.025219 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 3868
I20260808 01:49:35.025852 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:49:35.034576 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.68:0
--local_ip_for_outbound_sockets=127.29.180.68
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20260808 01:49:35.079022 4090 heartbeater.cc:344] Connected to a master server at 127.29.180.126:40305
I20260808 01:49:35.079605 4090 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:35.081034 4090 heartbeater.cc:507] Master 127.29.180.126:40305 requested a full tablet report, sending...
I20260808 01:49:35.083920 3352 ts_manager.cc:194] Registered new tserver with Master: cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:35.085394 3352 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:50411
W20260808 01:49:35.367338 4094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:35.367871 4094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:35.368301 4094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:35.403838 4094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:35.404666 4094 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.68
I20260808 01:49:35.443513 4094 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:40305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:35.444849 4094 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:35.446485 4094 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:35.459931 4100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:36.089761 4090 heartbeater.cc:499] Master 127.29.180.126:40305 was elected leader, sending a full tablet report...
W20260808 01:49:35.460585 4101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:36.781343 4103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:36.786242 4102 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1315 milliseconds
W20260808 01:49:36.786929 4094 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.327s user 0.480s sys 0.837s
W20260808 01:49:36.787277 4094 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.327s user 0.480s sys 0.837s
I20260808 01:49:36.787567 4094 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:36.788813 4094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:36.791514 4094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:36.792972 4094 hybrid_clock.cc:648] HybridClock initialized: now 1786153776792941 us; error 46 us; skew 500 ppm
I20260808 01:49:36.793982 4094 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:36.801928 4094 webserver.cc:533] Webserver started at http://127.29.180.68:36451/ using document root <none> and password file <none>
I20260808 01:49:36.803949 4094 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:36.804266 4094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:36.804984 4094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:36.809713 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:36.810984 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-1/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:36.811959 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-2/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:36.812884 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-3/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:36.814071 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-4/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:36.815003 4094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance:
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:37.430222 4094 fs_manager.cc:696] Time spent creating directory manager: real 0.615s user 0.256s sys 0.276s
I20260808 01:49:37.440660 4094 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260808 01:49:37.441030 4094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal
uuid: "65cad62c709c4692b82a107852079545"
format_stamp: "Formatted at 2026-08-08 01:49:36 on dist-test-slave-p3t5"
I20260808 01:49:37.441489 4094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal
5 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:37.497517 4094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:37.498963 4094 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:37.499437 4094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:37.501422 4094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:37.504899 4094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:37.505148 4094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:37.505398 4094 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:37.505569 4094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:37.648942 4094 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.68:39673
I20260808 01:49:37.649030 4317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.68:39673 every 8 connection(s)
I20260808 01:49:37.651530 4094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
I20260808 01:49:37.657196 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 4094
I20260808 01:49:37.657761 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance
I20260808 01:49:37.699160 4318 heartbeater.cc:344] Connected to a master server at 127.29.180.126:40305
I20260808 01:49:37.699693 4318 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:37.700809 4318 heartbeater.cc:507] Master 127.29.180.126:40305 requested a full tablet report, sending...
I20260808 01:49:37.703130 3352 ts_manager.cc:194] Registered new tserver with Master: 65cad62c709c4692b82a107852079545 (127.29.180.68:39673)
I20260808 01:49:37.704485 3352 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.68:50201
I20260808 01:49:37.709843 30417 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20260808 01:49:37.793936 3352 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53626:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 01:49:37.870882 4025 tablet_service.cc:1511] Processing CreateTablet for tablet 53c0974b72fa4e70ac5557334e7a22dc (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:37.872769 4024 tablet_service.cc:1511] Processing CreateTablet for tablet 372d004f79bd4380ad523050a5fc7828 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:37.878783 4023 tablet_service.cc:1511] Processing CreateTablet for tablet 7fd2197103a04caca6b06ab3514948c1 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:37.882073 4022 tablet_service.cc:1511] Processing CreateTablet for tablet 9c6d0981260e4364afaf2d268ce3af37 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:37.876286 4251 tablet_service.cc:1511] Processing CreateTablet for tablet d6a557ab499d480c9a56965375358874 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:37.882097 4250 tablet_service.cc:1511] Processing CreateTablet for tablet 7fd2197103a04caca6b06ab3514948c1 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:37.870290 4252 tablet_service.cc:1511] Processing CreateTablet for tablet 53c0974b72fa4e70ac5557334e7a22dc (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:37.913915 3795 tablet_service.cc:1511] Processing CreateTablet for tablet 372d004f79bd4380ad523050a5fc7828 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:37.918401 3796 tablet_service.cc:1511] Processing CreateTablet for tablet 53c0974b72fa4e70ac5557334e7a22dc (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION VALUES < 429496729
I20260808 01:49:37.911864 3793 tablet_service.cc:1511] Processing CreateTablet for tablet 9c6d0981260e4364afaf2d268ce3af37 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:37.924366 3794 tablet_service.cc:1511] Processing CreateTablet for tablet d6a557ab499d480c9a56965375358874 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:37.938709 3569 tablet_service.cc:1511] Processing CreateTablet for tablet 7fd2197103a04caca6b06ab3514948c1 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916
I20260808 01:49:37.938868 3570 tablet_service.cc:1511] Processing CreateTablet for tablet d6a557ab499d480c9a56965375358874 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187
I20260808 01:49:37.939837 3571 tablet_service.cc:1511] Processing CreateTablet for tablet 372d004f79bd4380ad523050a5fc7828 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458
I20260808 01:49:37.950153 3567 tablet_service.cc:1511] Processing CreateTablet for tablet 9c6d0981260e4364afaf2d268ce3af37 (DEFAULT_TABLE table=test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]), partition=RANGE (key) PARTITION 1717986916 <= VALUES
I20260808 01:49:37.987686 4346 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: Bootstrap starting.
I20260808 01:49:37.994588 4344 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: Bootstrap starting.
I20260808 01:49:38.002954 4344 tablet_bootstrap.cc:654] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.005676 4344 log.cc:826] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:38.011319 4347 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5: Bootstrap starting.
I20260808 01:49:38.015266 4345 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: Bootstrap starting.
I20260808 01:49:38.041224 4346 tablet_bootstrap.cc:654] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.044943 4348 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: Bootstrap starting.
I20260808 01:49:38.056592 4351 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: Bootstrap starting.
I20260808 01:49:38.062299 4345 tablet_bootstrap.cc:654] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.079541 4347 tablet_bootstrap.cc:654] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.092497 4344 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: No bootstrap required, opened a new log
I20260808 01:49:38.096078 4355 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:38.096969 4344 ts_tablet_manager.cc:1403] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: Time spent bootstrapping tablet: real 0.103s user 0.041s sys 0.053s
I20260808 01:49:38.106129 4346 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: No bootstrap required, opened a new log
I20260808 01:49:38.113147 4346 ts_tablet_manager.cc:1403] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: Time spent bootstrapping tablet: real 0.126s user 0.016s sys 0.007s
I20260808 01:49:38.118094 4356 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap starting.
I20260808 01:49:38.122838 4357 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:38.131819 4345 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: No bootstrap required, opened a new log
I20260808 01:49:38.132571 4345 ts_tablet_manager.cc:1403] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: Time spent bootstrapping tablet: real 0.118s user 0.020s sys 0.037s
I20260808 01:49:38.135519 4362 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Bootstrap starting.
I20260808 01:49:38.086957 4350 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545: Bootstrap starting.
I20260808 01:49:38.094272 4351 tablet_bootstrap.cc:654] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.076381 4348 tablet_bootstrap.cc:654] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.139554 4351 log.cc:826] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:38.147675 4359 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:38.135718 4355 tablet_bootstrap.cc:654] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.153079 4346 raft_consensus.cc:359] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.154031 4346 raft_consensus.cc:385] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.154340 4346 raft_consensus.cc:740] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.155274 4346 consensus_queue.cc:260] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.156764 4358 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap starting.
I20260808 01:49:38.157030 4345 raft_consensus.cc:359] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.158553 4345 raft_consensus.cc:385] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.158934 4345 raft_consensus.cc:740] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.159343 4355 log.cc:826] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:38.159862 4345 consensus_queue.cc:260] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.161347 4360 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap starting.
I20260808 01:49:38.167291 4361 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:38.171212 4344 raft_consensus.cc:359] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.172263 4344 raft_consensus.cc:385] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.172636 4344 raft_consensus.cc:740] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.173084 4351 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: No bootstrap required, opened a new log
I20260808 01:49:38.173641 4344 consensus_queue.cc:260] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.175653 4362 tablet_bootstrap.cc:654] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.175832 4347 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5: No bootstrap required, opened a new log
I20260808 01:49:38.176260 4347 ts_tablet_manager.cc:1403] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5: Time spent bootstrapping tablet: real 0.165s user 0.007s sys 0.007s
I20260808 01:49:38.178077 4355 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: No bootstrap required, opened a new log
I20260808 01:49:38.178905 4351 ts_tablet_manager.cc:1403] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: Time spent bootstrapping tablet: real 0.123s user 0.014s sys 0.011s
I20260808 01:49:38.179152 4362 log.cc:826] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:38.182989 4355 ts_tablet_manager.cc:1403] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.000s
I20260808 01:49:38.183365 4345 ts_tablet_manager.cc:1434] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: Time spent starting tablet: real 0.050s user 0.001s sys 0.009s
I20260808 01:49:38.217056 4362 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: No bootstrap required, opened a new log
I20260808 01:49:38.218477 4362 ts_tablet_manager.cc:1403] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.015s
I20260808 01:49:38.260344 4366 raft_consensus.cc:493] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.261579 4366 raft_consensus.cc:515] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.280896 4350 tablet_bootstrap.cc:654] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:38.283229 4091 tablet.cc:2406] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:38.282853 4366 leader_election.cc:290] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.299072 4355 raft_consensus.cc:359] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.300812 4355 raft_consensus.cc:385] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.301543 4355 raft_consensus.cc:740] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.303224 4344 ts_tablet_manager.cc:1434] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5: Time spent starting tablet: real 0.206s user 0.058s sys 0.074s
I20260808 01:49:38.303855 4355 consensus_queue.cc:260] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.307085 4350 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545: No bootstrap required, opened a new log
I20260808 01:49:38.307716 4350 ts_tablet_manager.cc:1403] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545: Time spent bootstrapping tablet: real 0.221s user 0.016s sys 0.003s
I20260808 01:49:38.322841 4360 tablet_bootstrap.cc:654] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.225561 4347 raft_consensus.cc:359] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.349267 4347 raft_consensus.cc:385] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.350106 4347 raft_consensus.cc:740] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.351855 4359 tablet_bootstrap.cc:654] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.351929 4347 consensus_queue.cc:260] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.361011 4346 ts_tablet_manager.cc:1434] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: Time spent starting tablet: real 0.245s user 0.086s sys 0.089s
I20260808 01:49:38.366489 4347 ts_tablet_manager.cc:1434] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5: Time spent starting tablet: real 0.190s user 0.005s sys 0.008s
I20260808 01:49:38.356177 4348 tablet_bootstrap.cc:492] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: No bootstrap required, opened a new log
I20260808 01:49:38.374464 4348 ts_tablet_manager.cc:1403] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: Time spent bootstrapping tablet: real 0.340s user 0.009s sys 0.009s
I20260808 01:49:38.373219 4373 raft_consensus.cc:493] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.395772 4355 ts_tablet_manager.cc:1434] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.204s user 0.044s sys 0.026s
I20260808 01:49:38.387975 4373 raft_consensus.cc:515] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.401875 3816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
W20260808 01:49:38.406497 3977 leader_election.cc:343] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:38.411736 4358 tablet_bootstrap.cc:654] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.414520 4348 raft_consensus.cc:359] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.415509 4348 raft_consensus.cc:385] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.415808 4348 raft_consensus.cc:740] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cad62c709c4692b82a107852079545, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.417107 4348 consensus_queue.cc:260] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.419509 4351 raft_consensus.cc:359] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.420286 4351 raft_consensus.cc:385] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.420643 4351 raft_consensus.cc:740] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cad62c709c4692b82a107852079545, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.421597 4351 consensus_queue.cc:260] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.424557 4357 tablet_bootstrap.cc:654] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.425417 4350 raft_consensus.cc:359] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.427032 4350 raft_consensus.cc:385] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.428526 4350 raft_consensus.cc:740] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cad62c709c4692b82a107852079545, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.437045 4361 tablet_bootstrap.cc:654] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.410286 4356 tablet_bootstrap.cc:654] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:38.430998 4350 consensus_queue.cc:260] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
W20260808 01:49:38.448769 3638 tablet.cc:2406] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:38.460354 3591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:38.461402 4362 raft_consensus.cc:359] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.462797 4362 raft_consensus.cc:385] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.463195 4362 raft_consensus.cc:740] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Initialized, Role: FOLLOWER
W20260808 01:49:38.466226 3977 leader_election.cc:343] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:38.467162 3977 leader_election.cc:304] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc7f064562454e9586acf4bc50ced2f5; no voters: 529e440fab074f228c5eee925f4a3bb5, 9274dcc435444a69a98458b1d37e28a4
I20260808 01:49:38.470898 4364 raft_consensus.cc:2749] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:38.488910 4360 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5: No bootstrap required, opened a new log
I20260808 01:49:38.489503 4361 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: No bootstrap required, opened a new log
I20260808 01:49:38.489609 4360 ts_tablet_manager.cc:1403] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5: Time spent bootstrapping tablet: real 0.329s user 0.000s sys 0.032s
I20260808 01:49:38.490278 4361 ts_tablet_manager.cc:1403] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.323s user 0.015s sys 0.006s
I20260808 01:49:38.490696 4350 ts_tablet_manager.cc:1434] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545: Time spent starting tablet: real 0.181s user 0.024s sys 0.009s
I20260808 01:49:38.499334 4379 raft_consensus.cc:493] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.500396 4379 raft_consensus.cc:515] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.505082 4379 leader_election.cc:290] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.496074 4361 raft_consensus.cc:359] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.518231 4361 raft_consensus.cc:385] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.514262 4360 raft_consensus.cc:359] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.518778 4361 raft_consensus.cc:740] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.518748 4360 raft_consensus.cc:385] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.519246 4360 raft_consensus.cc:740] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.514688 4362 consensus_queue.cc:260] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.520985 4361 consensus_queue.cc:260] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.535440 4360 consensus_queue.cc:260] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.544622 4364 raft_consensus.cc:493] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.545475 4364 raft_consensus.cc:515] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.551789 4362 ts_tablet_manager.cc:1434] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Time spent starting tablet: real 0.325s user 0.038s sys 0.025s
I20260808 01:49:38.559974 4364 leader_election.cc:290] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65cad62c709c4692b82a107852079545 (127.29.180.68:39673), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.565599 4364 raft_consensus.cc:493] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.566349 4364 raft_consensus.cc:515] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.571409 3591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:38.572470 3591 raft_consensus.cc:2468] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 0.
I20260808 01:49:38.574270 4368 raft_consensus.cc:493] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.575244 3977 leader_election.cc:304] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9274dcc435444a69a98458b1d37e28a4, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:38.575395 4368 raft_consensus.cc:515] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.576647 4366 raft_consensus.cc:2804] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:38.577097 4366 raft_consensus.cc:493] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:38.577943 4366 raft_consensus.cc:3060] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.587891 4368 leader_election.cc:290] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.587960 3591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:38.588711 3591 raft_consensus.cc:2468] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 0.
I20260808 01:49:38.591066 4364 leader_election.cc:290] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 65cad62c709c4692b82a107852079545 (127.29.180.68:39673)
I20260808 01:49:38.527956 4373 leader_election.cc:290] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737), cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:38.559131 3816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
I20260808 01:49:38.593885 3977 leader_election.cc:304] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9274dcc435444a69a98458b1d37e28a4, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:38.597993 4368 raft_consensus.cc:2804] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:38.598713 4368 raft_consensus.cc:493] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:38.599426 4368 raft_consensus.cc:3060] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.508989 4380 raft_consensus.cc:493] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.602164 4380 raft_consensus.cc:515] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.603550 4366 raft_consensus.cc:515] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.607618 4366 leader_election.cc:290] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 65cad62c709c4692b82a107852079545 (127.29.180.68:39673)
I20260808 01:49:38.608042 4318 heartbeater.cc:499] Master 127.29.180.126:40305 was elected leader, sending a full tablet report...
I20260808 01:49:38.524318 4351 ts_tablet_manager.cc:1434] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545: Time spent starting tablet: real 0.345s user 0.028s sys 0.014s
I20260808 01:49:38.615258 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4"
I20260808 01:49:38.616420 3590 raft_consensus.cc:3060] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.618453 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "9274dcc435444a69a98458b1d37e28a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5" is_pre_election: true
I20260808 01:49:38.564450 4348 ts_tablet_manager.cc:1434] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: Time spent starting tablet: real 0.189s user 0.009s sys 0.010s
I20260808 01:49:38.625921 4380 leader_election.cc:290] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.636168 4361 ts_tablet_manager.cc:1434] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.145s user 0.009s sys 0.003s
I20260808 01:49:38.643600 3590 raft_consensus.cc:2468] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:38.645875 4359 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: No bootstrap required, opened a new log
I20260808 01:49:38.646543 4359 ts_tablet_manager.cc:1403] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.499s user 0.022s sys 0.005s
I20260808 01:49:38.649557 3977 leader_election.cc:304] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9274dcc435444a69a98458b1d37e28a4, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:38.653560 4359 raft_consensus.cc:359] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.656242 4359 raft_consensus.cc:385] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 01:49:38.658914 4319 tablet.cc:2406] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:38.659760 4360 ts_tablet_manager.cc:1434] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5: Time spent starting tablet: real 0.170s user 0.015s sys 0.003s
I20260808 01:49:38.661576 4359 raft_consensus.cc:740] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.663123 4359 consensus_queue.cc:260] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.591761 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
I20260808 01:49:38.667760 4359 ts_tablet_manager.cc:1434] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.021s user 0.007s sys 0.004s
I20260808 01:49:38.668440 4366 raft_consensus.cc:2804] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:38.635869 3816 raft_consensus.cc:2468] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 0.
I20260808 01:49:38.669340 4366 raft_consensus.cc:697] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Becoming Leader. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Running, Role: LEADER
I20260808 01:49:38.670821 4366 consensus_queue.cc:237] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.671517 4390 raft_consensus.cc:493] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.672567 4368 raft_consensus.cc:515] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.674535 4045 raft_consensus.cc:2393] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9274dcc435444a69a98458b1d37e28a4 in current term 1: Already voted for candidate cc7f064562454e9586acf4bc50ced2f5 in this term.
I20260808 01:49:38.672658 4390 raft_consensus.cc:515] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.677269 3815 raft_consensus.cc:2468] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 0.
I20260808 01:49:38.686908 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4"
I20260808 01:49:38.668488 3814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "9274dcc435444a69a98458b1d37e28a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
I20260808 01:49:38.687659 3590 raft_consensus.cc:3060] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.693526 3977 leader_election.cc:304] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, cc7f064562454e9586acf4bc50ced2f5; no voters:
W20260808 01:49:38.695858 3865 tablet.cc:2406] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:38.699335 3524 leader_election.cc:304] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9274dcc435444a69a98458b1d37e28a4; no voters: 529e440fab074f228c5eee925f4a3bb5, cc7f064562454e9586acf4bc50ced2f5
I20260808 01:49:38.710911 4358 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5: No bootstrap required, opened a new log
I20260808 01:49:38.711746 4358 ts_tablet_manager.cc:1403] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5: Time spent bootstrapping tablet: real 0.555s user 0.026s sys 0.059s
I20260808 01:49:38.726186 4356 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: No bootstrap required, opened a new log
I20260808 01:49:38.727020 4356 ts_tablet_manager.cc:1403] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: Time spent bootstrapping tablet: real 0.610s user 0.029s sys 0.052s
I20260808 01:49:38.729599 3590 raft_consensus.cc:2468] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:38.720664 4358 raft_consensus.cc:359] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.734194 4358 raft_consensus.cc:385] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.734558 4358 raft_consensus.cc:740] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.737843 4358 consensus_queue.cc:260] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.751776 4356 raft_consensus.cc:359] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.764074 4357 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: No bootstrap required, opened a new log
I20260808 01:49:38.764926 4357 ts_tablet_manager.cc:1403] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.642s user 0.015s sys 0.009s
I20260808 01:49:38.766429 4373 raft_consensus.cc:2749] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:49:38.766742 3524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.180.66:33737, user_credentials={real_user=slave}} blocked reactor thread for 68080us
I20260808 01:49:38.694765 4364 raft_consensus.cc:2804] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:38.754995 4356 raft_consensus.cc:385] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.768721 4364 raft_consensus.cc:493] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:38.768944 4356 raft_consensus.cc:740] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.769338 4364 raft_consensus.cc:3060] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.772262 4368 leader_election.cc:290] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.783775 4356 consensus_queue.cc:260] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.774104 4390 leader_election.cc:290] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221), 65cad62c709c4692b82a107852079545 (127.29.180.68:39673)
W20260808 01:49:38.797724 3977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.180.66:33737, user_credentials={real_user=slave}} blocked reactor thread for 104426us
I20260808 01:49:38.805506 4380 raft_consensus.cc:493] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:38.806490 4380 raft_consensus.cc:515] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:38.808960 4358 ts_tablet_manager.cc:1434] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5: Time spent starting tablet: real 0.097s user 0.004s sys 0.013s
I20260808 01:49:38.811928 4380 leader_election.cc:290] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:38.793099 4356 ts_tablet_manager.cc:1434] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: Time spent starting tablet: real 0.066s user 0.004s sys 0.008s
I20260808 01:49:38.814797 3814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c6d0981260e4364afaf2d268ce3af37" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5"
I20260808 01:49:38.815755 3814 raft_consensus.cc:3060] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.820293 3351 catalog_manager.cc:5697] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:38.836184 4357 raft_consensus.cc:359] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.836946 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6a557ab499d480c9a56965375358874" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
I20260808 01:49:38.838073 4357 raft_consensus.cc:385] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:38.838100 3815 raft_consensus.cc:2468] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65cad62c709c4692b82a107852079545 in term 0.
I20260808 01:49:38.838672 4357 raft_consensus.cc:740] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:38.840728 4357 consensus_queue.cc:260] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.842564 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5" is_pre_election: true
I20260808 01:49:38.820421 3977 leader_election.cc:304] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9274dcc435444a69a98458b1d37e28a4, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:38.843340 3815 raft_consensus.cc:2468] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65cad62c709c4692b82a107852079545 in term 0.
I20260808 01:49:38.846005 4366 raft_consensus.cc:2804] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:38.846309 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6a557ab499d480c9a56965375358874" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:38.846662 3591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:38.846973 4366 raft_consensus.cc:697] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Becoming Leader. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Running, Role: LEADER
I20260808 01:49:38.847644 3591 raft_consensus.cc:2393] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65cad62c709c4692b82a107852079545 in current term 1: Already voted for candidate cc7f064562454e9586acf4bc50ced2f5 in this term.
I20260808 01:49:38.847460 3590 raft_consensus.cc:2468] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65cad62c709c4692b82a107852079545 in term 0.
I20260808 01:49:38.851342 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5" is_pre_election: true
I20260808 01:49:38.848778 4366 consensus_queue.cc:237] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:38.853430 4357 ts_tablet_manager.cc:1434] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.088s user 0.012s sys 0.008s
I20260808 01:49:38.854985 4205 leader_election.cc:304] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, 65cad62c709c4692b82a107852079545; no voters:
I20260808 01:49:38.856540 4044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5" is_pre_election: true
I20260808 01:49:38.861836 4364 raft_consensus.cc:515] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.864701 4045 raft_consensus.cc:2393] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65cad62c709c4692b82a107852079545 in current term 1: Already voted for candidate cc7f064562454e9586acf4bc50ced2f5 in this term.
I20260808 01:49:38.866899 4380 raft_consensus.cc:2804] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:38.867633 4380 raft_consensus.cc:493] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:38.868347 4380 raft_consensus.cc:3060] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.866925 4364 leader_election.cc:290] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Requested vote from peers 65cad62c709c4692b82a107852079545 (127.29.180.68:39673), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.878659 4205 leader_election.cc:304] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, 65cad62c709c4692b82a107852079545; no voters: cc7f064562454e9586acf4bc50ced2f5
I20260808 01:49:38.881472 4379 raft_consensus.cc:2804] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:38.882112 4379 raft_consensus.cc:493] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:38.881762 4205 leader_election.cc:304] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65cad62c709c4692b82a107852079545; no voters: 9274dcc435444a69a98458b1d37e28a4, cc7f064562454e9586acf4bc50ced2f5
I20260808 01:49:38.882675 4379 raft_consensus.cc:3060] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.884476 4377 raft_consensus.cc:3060] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.884480 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5"
I20260808 01:49:38.885397 3815 raft_consensus.cc:3060] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.896739 3814 raft_consensus.cc:2468] T 9c6d0981260e4364afaf2d268ce3af37 P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:38.902262 3815 raft_consensus.cc:2468] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:38.904495 3977 leader_election.cc:304] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:38.906316 4366 raft_consensus.cc:2804] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:38.907331 4366 raft_consensus.cc:697] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Becoming Leader. State: Replica: cc7f064562454e9586acf4bc50ced2f5, State: Running, Role: LEADER
I20260808 01:49:38.909025 4366 consensus_queue.cc:237] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.928314 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5" is_pre_election: true
I20260808 01:49:38.942631 4377 raft_consensus.cc:2749] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:38.918573 4380 raft_consensus.cc:515] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.949651 4272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cad62c709c4692b82a107852079545" is_pre_election: true
I20260808 01:49:38.956399 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6a557ab499d480c9a56965375358874" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4"
I20260808 01:49:38.948386 3352 catalog_manager.cc:5697] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:38.957492 3590 raft_consensus.cc:3060] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.958587 3352 catalog_manager.cc:5697] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:38.953447 4379 raft_consensus.cc:515] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:38.964552 4380 leader_election.cc:290] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 election: Requested vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.967105 4272 raft_consensus.cc:2393] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 529e440fab074f228c5eee925f4a3bb5 in current term 1: Already voted for candidate 65cad62c709c4692b82a107852079545 in this term.
I20260808 01:49:38.973117 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5"
I20260808 01:49:38.974331 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6a557ab499d480c9a56965375358874" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5"
I20260808 01:49:38.975231 3815 raft_consensus.cc:3060] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:38.984043 3590 raft_consensus.cc:2468] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65cad62c709c4692b82a107852079545 in term 1.
I20260808 01:49:38.983655 4271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cad62c709c4692b82a107852079545" is_pre_election: true
I20260808 01:49:38.988931 3815 raft_consensus.cc:2468] T d6a557ab499d480c9a56965375358874 P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65cad62c709c4692b82a107852079545 in term 1.
I20260808 01:49:38.993146 4379 leader_election.cc:290] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 election: Requested vote from peers cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221), 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:38.995565 4271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cad62c709c4692b82a107852079545" is_pre_election: true
I20260808 01:49:38.997007 4271 raft_consensus.cc:2468] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:39.000839 3752 leader_election.cc:304] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5; no voters: 65cad62c709c4692b82a107852079545, cc7f064562454e9586acf4bc50ced2f5
I20260808 01:49:39.002542 4390 raft_consensus.cc:2749] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:39.000499 4269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cad62c709c4692b82a107852079545"
I20260808 01:49:39.003690 4269 raft_consensus.cc:2393] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cc7f064562454e9586acf4bc50ced2f5 in current term 1: Already voted for candidate 65cad62c709c4692b82a107852079545 in this term.
I20260808 01:49:38.997407 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fd2197103a04caca6b06ab3514948c1" candidate_uuid: "cc7f064562454e9586acf4bc50ced2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cad62c709c4692b82a107852079545"
I20260808 01:49:39.009595 4205 leader_election.cc:304] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, 65cad62c709c4692b82a107852079545; no voters:
I20260808 01:49:39.013605 4379 raft_consensus.cc:2804] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:39.020208 4270 raft_consensus.cc:2468] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc7f064562454e9586acf4bc50ced2f5 in term 1.
I20260808 01:49:39.021492 4379 raft_consensus.cc:697] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [term 1 LEADER]: Becoming Leader. State: Replica: 65cad62c709c4692b82a107852079545, State: Running, Role: LEADER
I20260808 01:49:39.023824 3815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53c0974b72fa4e70ac5557334e7a22dc" candidate_uuid: "65cad62c709c4692b82a107852079545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e440fab074f228c5eee925f4a3bb5"
I20260808 01:49:39.026530 4390 raft_consensus.cc:493] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:39.027654 3815 raft_consensus.cc:2393] T 53c0974b72fa4e70ac5557334e7a22dc P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65cad62c709c4692b82a107852079545 in current term 1: Already voted for candidate cc7f064562454e9586acf4bc50ced2f5 in this term.
I20260808 01:49:39.027462 4390 raft_consensus.cc:515] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:39.025799 4379 consensus_queue.cc:237] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:39.029783 4205 leader_election.cc:304] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65cad62c709c4692b82a107852079545; no voters: 529e440fab074f228c5eee925f4a3bb5, cc7f064562454e9586acf4bc50ced2f5
I20260808 01:49:39.032163 4380 raft_consensus.cc:2749] T 53c0974b72fa4e70ac5557334e7a22dc P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:39.051139 4390 leader_election.cc:290] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:39.052486 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5" is_pre_election: true
I20260808 01:49:39.053571 4045 raft_consensus.cc:2468] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529e440fab074f228c5eee925f4a3bb5 in term 0.
I20260808 01:49:39.058069 3751 leader_election.cc:304] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, cc7f064562454e9586acf4bc50ced2f5; no voters:
I20260808 01:49:39.059720 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4" is_pre_election: true
I20260808 01:49:39.060617 4391 raft_consensus.cc:2804] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:39.060807 3590 raft_consensus.cc:2468] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529e440fab074f228c5eee925f4a3bb5 in term 0.
I20260808 01:49:39.061375 4391 raft_consensus.cc:493] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:39.062150 4391 raft_consensus.cc:3060] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:39.076375 3352 catalog_manager.cc:5697] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65cad62c709c4692b82a107852079545 (127.29.180.68). New cstate: current_term: 1 leader_uuid: "65cad62c709c4692b82a107852079545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:39.079263 4391 raft_consensus.cc:515] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:39.084232 3590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9274dcc435444a69a98458b1d37e28a4"
I20260808 01:49:39.085353 3590 raft_consensus.cc:3060] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:39.096625 4045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "372d004f79bd4380ad523050a5fc7828" candidate_uuid: "529e440fab074f228c5eee925f4a3bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7f064562454e9586acf4bc50ced2f5"
I20260808 01:49:39.097466 4045 raft_consensus.cc:3060] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:39.103303 3590 raft_consensus.cc:2468] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529e440fab074f228c5eee925f4a3bb5 in term 1.
I20260808 01:49:39.105804 4391 leader_election.cc:290] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 election: Requested vote from peers 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729), cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:39.110211 3750 leader_election.cc:304] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529e440fab074f228c5eee925f4a3bb5, 9274dcc435444a69a98458b1d37e28a4; no voters:
I20260808 01:49:39.112504 4391 raft_consensus.cc:2804] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:39.113864 4045 raft_consensus.cc:2468] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529e440fab074f228c5eee925f4a3bb5 in term 1.
I20260808 01:49:39.115463 4391 raft_consensus.cc:697] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Running, Role: LEADER
I20260808 01:49:39.118059 4391 consensus_queue.cc:237] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:39.135819 3352 catalog_manager.cc:5697] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "529e440fab074f228c5eee925f4a3bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:39.237476 4366 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:39.255909 4395 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:39.322090 4395 consensus_queue.cc:1048] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:39.343660 4364 consensus_queue.cc:1048] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:39.344015 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 3413
W20260808 01:49:39.387967 3977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.180.65:44729 (error 108)
I20260808 01:49:39.391484 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--block_manager=logr
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:44729
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:40305
--webserver_port=36595
--webserver_interface=127.29.180.65
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--env_inject_eio=1.0
--crash_on_eio=false
--builtin_ntp_servers=127.29.180.84:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20260808 01:49:39.394572 3977 consensus_peers.cc:597] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:39.486676 4379 consensus_queue.cc:1048] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:39.490803 4364 consensus_queue.cc:1048] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:39.508222 4366 consensus_queue.cc:1048] T 53c0974b72fa4e70ac5557334e7a22dc P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:39.522993 4205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111)
I20260808 01:49:39.529008 4390 consensus_queue.cc:1048] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:39.541827 4205 consensus_peers.cc:597] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:39.552608 3750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111)
W20260808 01:49:39.563550 3750 consensus_peers.cc:597] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:39.726014 3977 consensus_peers.cc:597] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:39.786484 4425 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:39.787086 4425 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:39.787717 4425 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:39.823449 4425 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:39.824492 4425 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:49:39.867048 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--block_manager=logr
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:44729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=36595
--tserver_master_addrs=127.29.180.126:40305
--env_inject_eio=1
--env_inject_eio_globs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/**
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:39.868546 4425 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260808 01:49:39.869444 4425 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:49:39.869586 4425 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260808 01:49:39.870931 4425 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:39.886900 4442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:39.887696 4443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:41.289744 4441 debug-util.cc:398] Leaking SignalData structure 0x7b080002ede0 after lost signal to thread 4425
W20260808 01:49:41.640944 4425 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.753s user 0.612s sys 1.061s
W20260808 01:49:41.642851 4425 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.755s user 0.612s sys 1.061s
W20260808 01:49:41.643581 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:41.647061 4444 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1754 milliseconds
I20260808 01:49:41.647081 4425 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:41.648648 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:41.662315 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:41.663780 4425 hybrid_clock.cc:648] HybridClock initialized: now 1786153781663698 us; error 86 us; skew 500 ppm
I20260808 01:49:41.664743 4425 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:41.671890 4425 webserver.cc:533] Webserver started at http://127.29.180.65:36595/ using document root <none> and password file <none>
I20260808 01:49:41.673849 4425 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:41.674153 4425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20260808 01:49:41.676862 4425 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:49:41.682237 4425 env_posix.cc:431] I/O error, context: INJECTED FAILURE
E20260808 01:49:41.697999 4425 env_posix.cc:431] I/O error, context: INJECTED FAILURE
W20260808 01:49:41.776234 3750 consensus_peers.cc:597] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:49:41.834321 4425 dir_manager.cc:319] Skip initializing rocksdb instance for the non-healthy directory /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data
W20260808 01:49:41.862435 3977 consensus_peers.cc:597] T 9c6d0981260e4364afaf2d268ce3af37 P cc7f064562454e9586acf4bc50ced2f5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:49:41.875891 4205 consensus_peers.cc:597] T d6a557ab499d480c9a56965375358874 P 65cad62c709c4692b82a107852079545 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:49:42.024554 3977 consensus_peers.cc:597] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 -> Peer 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Couldn't send request to peer 9274dcc435444a69a98458b1d37e28a4. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:44729: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:49:42.154703 4425 fs_manager.cc:714] Time spent opening directory manager: real 0.474s user 0.244s sys 0.197s
E20260808 01:49:42.159330 4425 log_block_manager.cc:3241] Not using report from /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5)
I20260808 01:49:42.161945 4425 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260808 01:49:42.162310 4425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "9274dcc435444a69a98458b1d37e28a4"
format_stamp: "Formatted at 2026-08-08 01:49:28 on dist-test-slave-p3t5"
I20260808 01:49:42.162899 4425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal
4 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-4/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:42.224676 4425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:42.226259 4425 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:42.226789 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:42.228957 4425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:42.235105 4539 ts_tablet_manager.cc:548] Loading tablet metadata (0/4 complete)
W20260808 01:49:42.250216 4540 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 7fd2197103a04caca6b06ab3514948c1: could not find data dir with uuid 1df16c997b7b4fd6a6589de938d414ec
W20260808 01:49:42.257124 4542 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Not found: could not load data dir group for tablet 372d004f79bd4380ad523050a5fc7828: could not find data dir with uuid 1df16c997b7b4fd6a6589de938d414ec
I20260808 01:49:42.257875 4425 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20260808 01:49:42.258203 4425 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.025s user 0.011s sys 0.006s
I20260808 01:49:42.258555 4425 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
I20260808 01:49:42.268224 4542 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:42.282778 4539 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
I20260808 01:49:42.283082 4541 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
E20260808 01:49:42.284997 4541 ts_tablet_manager.cc:1425] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:42.286988 4541 ts_tablet_manager.cc:1403] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260808 01:49:42.287994 4541 tablet_replica.cc:333] stopping tablet replica
I20260808 01:49:42.288559 4541 raft_consensus.cc:2243] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:42.300361 4541 raft_consensus.cc:2272] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:42.307315 4425 ts_tablet_manager.cc:616] Registered 4 tablets
I20260808 01:49:42.307763 4425 ts_tablet_manager.cc:595] Time spent register tablets: real 0.049s user 0.033s sys 0.000s
I20260808 01:49:42.307976 4541 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap starting.
E20260808 01:49:42.308634 4541 ts_tablet_manager.cc:1425] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Tablet failed to bootstrap: Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 372d004f79bd4380ad523050a5fc7828
I20260808 01:49:42.309114 4541 ts_tablet_manager.cc:1403] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 01:49:42.309479 4541 tablet_replica.cc:333] stopping tablet replica
I20260808 01:49:42.309834 4541 raft_consensus.cc:2243] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:42.310308 4541 raft_consensus.cc:2272] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:42.338896 4542 log.cc:826] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:42.370991 4542 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:42.373119 4542 tablet_bootstrap.cc:492] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap complete.
I20260808 01:49:42.376890 4542 ts_tablet_manager.cc:1403] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.109s user 0.054s sys 0.043s
I20260808 01:49:42.382164 4539 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:42.383512 4539 tablet_bootstrap.cc:492] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Bootstrap complete.
I20260808 01:49:42.385571 4539 ts_tablet_manager.cc:1403] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Time spent bootstrapping tablet: real 0.103s user 0.054s sys 0.032s
I20260808 01:49:42.419307 4539 raft_consensus.cc:359] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:42.420708 4539 raft_consensus.cc:740] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:42.422377 4542 raft_consensus.cc:359] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:42.423182 4542 raft_consensus.cc:740] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9274dcc435444a69a98458b1d37e28a4, State: Initialized, Role: FOLLOWER
I20260808 01:49:42.424223 4542 consensus_queue.cc:260] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:42.426532 4539 consensus_queue.cc:260] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9274dcc435444a69a98458b1d37e28a4" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 44729 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } }
I20260808 01:49:42.429243 4539 ts_tablet_manager.cc:1434] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.043s user 0.012s sys 0.009s
I20260808 01:49:42.430567 4542 ts_tablet_manager.cc:1434] T 9c6d0981260e4364afaf2d268ce3af37 P 9274dcc435444a69a98458b1d37e28a4: Time spent starting tablet: real 0.053s user 0.024s sys 0.004s
I20260808 01:49:42.559690 4425 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:44729
I20260808 01:49:42.560508 4658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:44729 every 8 connection(s)
I20260808 01:49:42.563869 4425 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
W20260808 01:49:42.572523 4660 tablet.cc:2406] T d6a557ab499d480c9a56965375358874 P 9274dcc435444a69a98458b1d37e28a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:42.574324 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 4425
I20260808 01:49:42.612756 4659 heartbeater.cc:344] Connected to a master server at 127.29.180.126:40305
I20260808 01:49:42.613387 4659 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:42.613293 3977 consensus_queue.cc:579] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: The tablet replica hosted on peer 9274dcc435444a69a98458b1d37e28a4 has failed
I20260808 01:49:42.614998 4659 heartbeater.cc:507] Master 127.29.180.126:40305 requested a full tablet report, sending...
I20260808 01:49:42.619822 3352 ts_manager.cc:194] Re-registered known tserver with Master: 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729)
W20260808 01:49:42.622454 3352 catalog_manager.cc:5610] Tablet 372d004f79bd4380ad523050a5fc7828 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) has failed on TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 372d004f79bd4380ad523050a5fc7828
W20260808 01:49:42.623013 3352 catalog_manager.cc:5610] Tablet 7fd2197103a04caca6b06ab3514948c1 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) has failed on TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:42.626034 4045 consensus_queue.cc:237] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } }
I20260808 01:49:42.626289 3352 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:56305
I20260808 01:49:42.633278 4270 raft_consensus.cc:1275] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Refusing update from remote peer cc7f064562454e9586acf4bc50ced2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:42.634773 4545 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:42.642382 4395 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } }
I20260808 01:49:42.644301 4270 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } }
I20260808 01:49:42.654345 3338 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7fd2197103a04caca6b06ab3514948c1 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:42.657457 3352 catalog_manager.cc:5697] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 reported cstate change: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } }
I20260808 01:49:42.683045 4045 consensus_queue.cc:237] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } }
I20260808 01:49:42.691715 4270 raft_consensus.cc:1275] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Refusing update from remote peer cc7f064562454e9586acf4bc50ced2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:49:42.692250 3977 consensus_peers.cc:597] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 -> Peer 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737): Couldn't send request to peer 529e440fab074f228c5eee925f4a3bb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fd2197103a04caca6b06ab3514948c1. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:42.693542 4395 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:42.701803 4591 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fd2197103a04caca6b06ab3514948c1 with delete_type TABLET_DATA_TOMBSTONED (TS 9274dcc435444a69a98458b1d37e28a4 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50920
I20260808 01:49:42.701704 4395 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } } }
I20260808 01:49:42.703905 4270 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } } }
I20260808 01:49:42.712980 3338 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7fd2197103a04caca6b06ab3514948c1 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:49:42.718159 4680 ts_tablet_manager.cc:1916] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:42.719851 3352 catalog_manager.cc:5697] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: config changed from index 2 to 3, NON_VOTER 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) added. New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:42.739105 4680 ts_tablet_manager.cc:1929] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:42.739675 4680 log.cc:1199] T 7fd2197103a04caca6b06ab3514948c1 P 9274dcc435444a69a98458b1d37e28a4: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:42.742182 3337 catalog_manager.cc:5028] TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): tablet 7fd2197103a04caca6b06ab3514948c1 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) successfully deleted
I20260808 01:49:42.748062 3750 consensus_queue.cc:579] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: The tablet replica hosted on peer 9274dcc435444a69a98458b1d37e28a4 has failed
I20260808 01:49:42.756871 3814 consensus_queue.cc:237] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } }
I20260808 01:49:42.762585 4045 raft_consensus.cc:1275] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Refusing update from remote peer 529e440fab074f228c5eee925f4a3bb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:42.764210 4434 consensus_queue.cc:1048] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:42.772284 4681 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } }
I20260808 01:49:42.773571 4045 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } }
I20260808 01:49:42.783288 3337 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 372d004f79bd4380ad523050a5fc7828 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:42.783851 3352 catalog_manager.cc:5697] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: config changed from index -1 to 2, VOTER 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "529e440fab074f228c5eee925f4a3bb5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } }
I20260808 01:49:42.793000 4591 tablet_service.cc:1558] Processing DeleteTablet for tablet 372d004f79bd4380ad523050a5fc7828 with delete_type TABLET_DATA_TOMBSTONED (TS 9274dcc435444a69a98458b1d37e28a4 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:50920
I20260808 01:49:42.793888 4680 ts_tablet_manager.cc:1916] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:42.798220 3814 consensus_queue.cc:237] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } }
I20260808 01:49:42.804936 4045 raft_consensus.cc:1275] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Refusing update from remote peer 529e440fab074f228c5eee925f4a3bb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:49:42.805356 3752 consensus_peers.cc:597] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 -> Peer 65cad62c709c4692b82a107852079545 (127.29.180.68:39673): Couldn't send request to peer 65cad62c709c4692b82a107852079545. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 372d004f79bd4380ad523050a5fc7828. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:42.806623 4681 consensus_queue.cc:1048] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:42.809422 4680 ts_tablet_manager.cc:1929] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown
I20260808 01:49:42.809803 4680 log.cc:1199] T 372d004f79bd4380ad523050a5fc7828 P 9274dcc435444a69a98458b1d37e28a4: Deleting WAL directory at /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/wal/wals/372d004f79bd4380ad523050a5fc7828
I20260808 01:49:42.814123 4659 heartbeater.cc:499] Master 127.29.180.126:40305 was elected leader, sending a full tablet report...
I20260808 01:49:42.814754 3337 catalog_manager.cc:5028] TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): tablet 372d004f79bd4380ad523050a5fc7828 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) successfully deleted
I20260808 01:49:42.815960 4434 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 65cad62c709c4692b82a107852079545 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } } }
I20260808 01:49:42.817540 4044 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 65cad62c709c4692b82a107852079545 (127.29.180.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } } }
W20260808 01:49:42.823606 3351 catalog_manager.cc:5610] Tablet 372d004f79bd4380ad523050a5fc7828 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) has failed on TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 372d004f79bd4380ad523050a5fc7828
W20260808 01:49:42.824278 3351 catalog_manager.cc:5610] Tablet 7fd2197103a04caca6b06ab3514948c1 (table test-workload [id=1bfe0bdcad8d471b8694dbd9dc85b739]) has failed on TS 9274dcc435444a69a98458b1d37e28a4 (127.29.180.65:44729): Not found: error retrieving tablet data dir group (one or more data dirs may have been removed): could not find data dir group for tablet 7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:42.832654 3337 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 372d004f79bd4380ad523050a5fc7828 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:49:42.836441 3352 catalog_manager.cc:5697] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: config changed from index 2 to 3, NON_VOTER 65cad62c709c4692b82a107852079545 (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "529e440fab074f228c5eee925f4a3bb5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } } }
I20260808 01:49:43.062999 3794 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:43.063294 4251 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260808 01:49:43.089936 4024 tablet_service.cc:1467] Tablet server has 3 leaders and 0 scanners
I20260808 01:49:43.122488 4591 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:49:43.129878 4708 ts_tablet_manager.cc:933] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Initiating tablet copy from peer cc7f064562454e9586acf4bc50ced2f5 (127.29.180.67:41221)
I20260808 01:49:43.145198 4708 tablet_copy_client.cc:323] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:41221
I20260808 01:49:43.148212 4065 tablet_copy_service.cc:140] P cc7f064562454e9586acf4bc50ced2f5: Received BeginTabletCopySession request for tablet 7fd2197103a04caca6b06ab3514948c1 from peer 529e440fab074f228c5eee925f4a3bb5 ({username='slave'} at 127.29.180.66:41085)
I20260808 01:49:43.149034 4065 tablet_copy_service.cc:161] P cc7f064562454e9586acf4bc50ced2f5: Beginning new tablet copy session on tablet 7fd2197103a04caca6b06ab3514948c1 from peer 529e440fab074f228c5eee925f4a3bb5 at {username='slave'} at 127.29.180.66:41085: session id = 529e440fab074f228c5eee925f4a3bb5-7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:43.159986 4065 tablet_copy_source_session.cc:215] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:43.204747 4710 ts_tablet_manager.cc:933] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Initiating tablet copy from peer 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66:33737)
I20260808 01:49:43.217154 4710 tablet_copy_client.cc:323] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:33737
I20260808 01:49:43.219700 3837 tablet_copy_service.cc:140] P 529e440fab074f228c5eee925f4a3bb5: Received BeginTabletCopySession request for tablet 372d004f79bd4380ad523050a5fc7828 from peer 65cad62c709c4692b82a107852079545 ({username='slave'} at 127.29.180.68:45321)
I20260808 01:49:43.220353 3837 tablet_copy_service.cc:161] P 529e440fab074f228c5eee925f4a3bb5: Beginning new tablet copy session on tablet 372d004f79bd4380ad523050a5fc7828 from peer 65cad62c709c4692b82a107852079545 at {username='slave'} at 127.29.180.68:45321: session id = 65cad62c709c4692b82a107852079545-372d004f79bd4380ad523050a5fc7828
I20260808 01:49:43.225769 4708 tablet_copy_client.cc:806] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:43.226497 4708 tablet_copy_client.cc:670] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:43.230204 3837 tablet_copy_source_session.cc:215] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:43.231879 4708 tablet_copy_client.cc:538] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:43.263506 4708 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap starting.
I20260808 01:49:43.277774 4710 tablet_copy_client.cc:806] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:43.278460 4710 tablet_copy_client.cc:670] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:43.283552 4710 tablet_copy_client.cc:538] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:43.297173 4710 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Bootstrap starting.
I20260808 01:49:43.312357 4708 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:43.313457 4708 tablet_bootstrap.cc:492] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Bootstrap complete.
I20260808 01:49:43.314671 4708 ts_tablet_manager.cc:1403] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Time spent bootstrapping tablet: real 0.051s user 0.026s sys 0.001s
I20260808 01:49:43.318188 4708 raft_consensus.cc:359] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } }
I20260808 01:49:43.319042 4708 raft_consensus.cc:740] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 529e440fab074f228c5eee925f4a3bb5, State: Initialized, Role: LEARNER
I20260808 01:49:43.319752 4708 consensus_queue.cc:260] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: NON_VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: true } }
I20260808 01:49:43.322441 4708 ts_tablet_manager.cc:1434] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 01:49:43.324354 4065 tablet_copy_service.cc:342] P cc7f064562454e9586acf4bc50ced2f5: Request end of tablet copy session 529e440fab074f228c5eee925f4a3bb5-7fd2197103a04caca6b06ab3514948c1 received from {username='slave'} at 127.29.180.66:41085
I20260808 01:49:43.324802 4065 tablet_copy_service.cc:434] P cc7f064562454e9586acf4bc50ced2f5: ending tablet copy session 529e440fab074f228c5eee925f4a3bb5-7fd2197103a04caca6b06ab3514948c1 on tablet 7fd2197103a04caca6b06ab3514948c1 with peer 529e440fab074f228c5eee925f4a3bb5
I20260808 01:49:43.371587 4710 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:43.372777 4710 tablet_bootstrap.cc:492] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Bootstrap complete.
I20260808 01:49:43.373662 4710 ts_tablet_manager.cc:1403] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Time spent bootstrapping tablet: real 0.077s user 0.028s sys 0.046s
I20260808 01:49:43.377120 4710 raft_consensus.cc:359] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } }
I20260808 01:49:43.378592 4710 raft_consensus.cc:740] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65cad62c709c4692b82a107852079545, State: Initialized, Role: LEARNER
I20260808 01:49:43.379482 4710 consensus_queue.cc:260] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: true } }
I20260808 01:49:43.381937 4710 ts_tablet_manager.cc:1434] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260808 01:49:43.383939 3837 tablet_copy_service.cc:342] P 529e440fab074f228c5eee925f4a3bb5: Request end of tablet copy session 65cad62c709c4692b82a107852079545-372d004f79bd4380ad523050a5fc7828 received from {username='slave'} at 127.29.180.68:45321
I20260808 01:49:43.384395 3837 tablet_copy_service.cc:434] P 529e440fab074f228c5eee925f4a3bb5: ending tablet copy session 65cad62c709c4692b82a107852079545-372d004f79bd4380ad523050a5fc7828 on tablet 372d004f79bd4380ad523050a5fc7828 with peer 65cad62c709c4692b82a107852079545
I20260808 01:49:43.510301 3814 raft_consensus.cc:1217] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3a98afc236b54bde99ac36162e176b62 | 127.29.180.126:40305 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:32899 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
529e440fab074f228c5eee925f4a3bb5 | 127.29.180.66:33737 | HEALTHY | <none> | 1 | 0
65cad62c709c4692b82a107852079545 | 127.29.180.68:39673 | HEALTHY | <none> | 1 | 0
9274dcc435444a69a98458b1d37e28a4 | 127.29.180.65:44729 | HEALTHY | <none> | 0 | 0
cc7f064562454e9586acf4bc50ced2f5 | 127.29.180.67:41221 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
crash_on_eio | false | experimental,advanced | 127.29.180.65:44729
env_inject_eio | 1 | hidden | 127.29.180.65:44729
env_inject_eio_globs | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-1/** | hidden | 127.29.180.65:44729
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:44729
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:33737
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:41221
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:39673
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:44729
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:33737
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:41221
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_5.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:39673
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:32899 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 4
Median | 4
Third Quartile | 5
Maximum | 5
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 9274dcc435444a69a98458b1d37e28a4 | 127.29.180.65:44729 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 5
Replicas | 15
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260808 01:49:43.574082 30417 log_verifier.cc:126] Checking tablet 372d004f79bd4380ad523050a5fc7828
E20260808 01:49:43.633970 4655 env_posix.cc:431] I/O error, context: INJECTED FAILURE
I20260808 01:49:43.673692 4270 raft_consensus.cc:1217] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260808 01:49:43.707068 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 372d004f79bd4380ad523050a5fc7828
I20260808 01:49:43.707393 30417 log_verifier.cc:126] Checking tablet 53c0974b72fa4e70ac5557334e7a22dc
I20260808 01:49:43.780361 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 53c0974b72fa4e70ac5557334e7a22dc
I20260808 01:49:43.780656 30417 log_verifier.cc:126] Checking tablet 7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:43.888134 30417 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7fd2197103a04caca6b06ab3514948c1
I20260808 01:49:43.888371 30417 log_verifier.cc:126] Checking tablet 9c6d0981260e4364afaf2d268ce3af37
I20260808 01:49:43.961743 4715 raft_consensus.cc:1064] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5: attempting to promote NON_VOTER 529e440fab074f228c5eee925f4a3bb5 to VOTER
I20260808 01:49:43.964706 4715 consensus_queue.cc:237] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false } }
I20260808 01:49:43.974475 3814 raft_consensus.cc:1275] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEARNER]: Refusing update from remote peer cc7f064562454e9586acf4bc50ced2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:43.974910 4270 raft_consensus.cc:1275] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Refusing update from remote peer cc7f064562454e9586acf4bc50ced2f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:43.977171 4395 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:43.978317 4715 consensus_queue.cc:1048] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:43.991393 4714 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false } } }
I20260808 01:49:43.993129 4270 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false } } }
I20260808 01:49:43.996134 3814 raft_consensus.cc:2955] T 7fd2197103a04caca6b06ab3514948c1 P 529e440fab074f228c5eee925f4a3bb5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false } } }
I20260808 01:49:43.997416 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9c6d0981260e4364afaf2d268ce3af37
I20260808 01:49:43.997728 30417 log_verifier.cc:126] Checking tablet d6a557ab499d480c9a56965375358874
I20260808 01:49:44.019150 3350 catalog_manager.cc:5697] T 7fd2197103a04caca6b06ab3514948c1 P cc7f064562454e9586acf4bc50ced2f5 reported cstate change: config changed from index 3 to 4, 529e440fab074f228c5eee925f4a3bb5 (127.29.180.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cc7f064562454e9586acf4bc50ced2f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:44.064363 4684 raft_consensus.cc:1064] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5: attempting to promote NON_VOTER 65cad62c709c4692b82a107852079545 to VOTER
I20260808 01:49:44.067787 4684 consensus_queue.cc:237] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false } }
I20260808 01:49:44.076685 4270 raft_consensus.cc:1275] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [term 1 LEARNER]: Refusing update from remote peer 529e440fab074f228c5eee925f4a3bb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:44.078614 4044 raft_consensus.cc:1275] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Refusing update from remote peer 529e440fab074f228c5eee925f4a3bb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:44.079026 4684 consensus_queue.cc:1048] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:44.080921 4718 consensus_queue.cc:1048] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:44.104135 4684 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 65cad62c709c4692b82a107852079545 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false } } }
I20260808 01:49:44.108043 4269 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P 65cad62c709c4692b82a107852079545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 65cad62c709c4692b82a107852079545 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false } } }
I20260808 01:49:44.130304 30417 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d6a557ab499d480c9a56965375358874
I20260808 01:49:44.132061 3351 catalog_manager.cc:5697] T 372d004f79bd4380ad523050a5fc7828 P 529e440fab074f228c5eee925f4a3bb5 reported cstate change: config changed from index 3 to 4, 65cad62c709c4692b82a107852079545 (127.29.180.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "529e440fab074f228c5eee925f4a3bb5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:44.135651 4044 raft_consensus.cc:2955] T 372d004f79bd4380ad523050a5fc7828 P cc7f064562454e9586acf4bc50ced2f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 65cad62c709c4692b82a107852079545 (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "529e440fab074f228c5eee925f4a3bb5" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 33737 } } peers { permanent_uuid: "cc7f064562454e9586acf4bc50ced2f5" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 41221 } } peers { permanent_uuid: "65cad62c709c4692b82a107852079545" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 39673 } attrs { promote: false } } }
I20260808 01:49:44.703342 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 4425
I20260808 01:49:44.806546 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 3641
W20260808 01:49:44.856777 3977 connection.cc:573] client connection to 127.29.180.66:33737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 01:49:44.857244 3977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 12 similar messages]
I20260808 01:49:44.868759 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 3868
I20260808 01:49:44.916255 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 4094
I20260808 01:49:44.973755 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 3228
2026-08-08T01:49:45Z chronyd exiting
[ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/5 (21858 ms)
[----------] 6 tests from DiskFailure/DiskFailureITest (116277 ms total)
[----------] 6 tests from TabletServerDiskError/TabletServerDiskErrorITest
[ RUN ] TabletServerDiskError/TabletServerDiskErrorITest.TestFailOnBootstrap/0
2026-08-08T01:49:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-08T01:49:45Z Disabled control of system clock
I20260808 01:49:45.129019 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:32885
--webserver_interface=127.29.180.126
--webserver_port=0
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.180.126:32885
--master_tombstone_evicted_tablet_replicas=false with env {}
W20260808 01:49:45.466459 4748 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:45.467177 4748 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:45.467684 4748 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:45.502887 4748 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 01:49:45.503269 4748 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:45.503572 4748 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 01:49:45.503845 4748 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260808 01:49:45.543642 4748 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.29.180.126:32885
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.180.126:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
--webserver_interface=127.29.180.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:45.545169 4748 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:45.547072 4748 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:45.560119 4754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:46.963400 4753 debug-util.cc:398] Leaking SignalData structure 0x7b08000290e0 after lost signal to thread 4748
W20260808 01:49:47.417685 4748 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.857s user 0.639s sys 1.217s
W20260808 01:49:47.418161 4748 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.858s user 0.639s sys 1.217s
W20260808 01:49:45.561033 4755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:47.419409 4756 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1856 milliseconds
W20260808 01:49:47.420809 4757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:47.420735 4748 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:47.424237 4748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:47.426968 4748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:47.428388 4748 hybrid_clock.cc:648] HybridClock initialized: now 1786153787428351 us; error 37 us; skew 500 ppm
I20260808 01:49:47.429291 4748 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:47.436695 4748 webserver.cc:533] Webserver started at http://127.29.180.126:45883/ using document root <none> and password file <none>
I20260808 01:49:47.438127 4748 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:47.438412 4748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:47.439031 4748 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:47.443769 4748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/instance:
uuid: "7704560c94d24a499a485f316d701990"
format_stamp: "Formatted at 2026-08-08 01:49:47 on dist-test-slave-p3t5"
I20260808 01:49:47.444967 4748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-1/instance:
uuid: "7704560c94d24a499a485f316d701990"
format_stamp: "Formatted at 2026-08-08 01:49:47 on dist-test-slave-p3t5"
I20260808 01:49:47.445989 4748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-2/instance:
uuid: "7704560c94d24a499a485f316d701990"
format_stamp: "Formatted at 2026-08-08 01:49:47 on dist-test-slave-p3t5"
I20260808 01:49:47.446921 4748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal/instance:
uuid: "7704560c94d24a499a485f316d701990"
format_stamp: "Formatted at 2026-08-08 01:49:47 on dist-test-slave-p3t5"
I20260808 01:49:47.463045 4748 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.010s sys 0.006s
I20260808 01:49:47.470487 4768 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:47.471256 4768 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:47.471829 4768 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:47.473281 4748 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260808 01:49:47.473600 4748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal
uuid: "7704560c94d24a499a485f316d701990"
format_stamp: "Formatted at 2026-08-08 01:49:47 on dist-test-slave-p3t5"
I20260808 01:49:47.474031 4748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:47.536700 4748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:47.538376 4748 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:47.538826 4748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:47.614501 4748 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.126:32885
I20260808 01:49:47.614598 4822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.126:32885 every 8 connection(s)
I20260808 01:49:47.617671 4748 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb
I20260808 01:49:47.627942 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 4748
I20260808 01:49:47.628439 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/wal/instance
I20260808 01:49:47.648922 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990: Bootstrap starting.
I20260808 01:49:47.655823 4823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:47.657702 4823 log.cc:826] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:47.663476 4823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990: No bootstrap required, opened a new log
I20260808 01:49:47.681265 4823 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } }
I20260808 01:49:47.682106 4823 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:47.682379 4823 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7704560c94d24a499a485f316d701990, State: Initialized, Role: FOLLOWER
I20260808 01:49:47.683151 4823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } }
I20260808 01:49:47.683743 4823 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:49:47.684044 4823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:49:47.684388 4823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:47.688759 4823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } }
I20260808 01:49:47.689518 4823 leader_election.cc:304] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7704560c94d24a499a485f316d701990; no voters:
I20260808 01:49:47.691318 4823 leader_election.cc:290] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 01:49:47.691622 4828 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:47.695310 4828 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [term 1 LEADER]: Becoming Leader. State: Replica: 7704560c94d24a499a485f316d701990, State: Running, Role: LEADER
I20260808 01:49:47.696146 4828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } }
I20260808 01:49:47.696518 4823 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:49:47.706710 4829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7704560c94d24a499a485f316d701990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } } }
I20260808 01:49:47.707530 4829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:47.708745 4830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7704560c94d24a499a485f316d701990. Latest consensus state: current_term: 1 leader_uuid: "7704560c94d24a499a485f316d701990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7704560c94d24a499a485f316d701990" member_type: VOTER last_known_addr { host: "127.29.180.126" port: 32885 } } }
I20260808 01:49:47.709589 4830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990 [sys.catalog]: This master's current role is: LEADER
I20260808 01:49:47.712002 4837 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:49:47.724704 4837 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:49:47.740796 4837 catalog_manager.cc:1383] Generated new cluster ID: 4e1e423d6999480e9a609d85eb6180be
I20260808 01:49:47.741150 4837 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:49:47.758594 4837 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:49:47.760716 4837 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:49:47.774770 4837 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7704560c94d24a499a485f316d701990: Generated new TSK 0
I20260808 01:49:47.775712 4837 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:49:47.799604 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:0
--local_ip_for_outbound_sockets=127.29.180.65
--webserver_interface=127.29.180.65
--webserver_port=0
--tserver_master_addrs=127.29.180.126:32885
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
W20260808 01:49:48.131345 4847 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:48.131934 4847 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:48.132517 4847 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:48.167958 4847 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:48.168563 4847 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260808 01:49:48.168854 4847 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260808 01:49:48.169384 4847 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:49:48.207926 4847 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.29.180.126:32885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:48.209450 4847 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:48.211364 4847 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:48.225492 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:48.229861 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:48.230775 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:49.474771 4855 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260808 01:49:49.475149 4847 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:49.479794 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:49.482926 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:49.484442 4847 hybrid_clock.cc:648] HybridClock initialized: now 1786153789484397 us; error 48 us; skew 500 ppm
I20260808 01:49:49.485594 4847 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:49.493716 4847 webserver.cc:533] Webserver started at http://127.29.180.65:44827/ using document root <none> and password file <none>
I20260808 01:49:49.495816 4847 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:49.496186 4847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:49.496960 4847 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:49.503633 4847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/instance:
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:49:49.505251 4847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/instance:
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:49:49.506523 4847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/instance:
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:49:49.507726 4847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance:
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:49:49.528303 4847 fs_manager.cc:696] Time spent creating directory manager: real 0.020s user 0.014s sys 0.007s
I20260808 01:49:49.538003 4866 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:49.538798 4866 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:49.539469 4866 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:49.541033 4847 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260808 01:49:49.541440 4847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:49:49.541983 4847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:49.624819 4847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:49.626405 4847 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:49.626856 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:49.630033 4847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:49.634588 4847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:49.634799 4847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:49.635015 4847 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:49.635164 4847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:49.780495 4847 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:33111
I20260808 01:49:49.780635 4982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:33111 every 8 connection(s)
I20260808 01:49:49.783530 4847 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:49:49.786439 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 4847
I20260808 01:49:49.787319 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal/instance
I20260808 01:49:49.797379 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.66:0
--local_ip_for_outbound_sockets=127.29.180.66
--webserver_interface=127.29.180.66
--webserver_port=0
--tserver_master_addrs=127.29.180.126:32885
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
I20260808 01:49:49.817341 4983 heartbeater.cc:344] Connected to a master server at 127.29.180.126:32885
I20260808 01:49:49.818066 4983 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:49.819908 4983 heartbeater.cc:507] Master 127.29.180.126:32885 requested a full tablet report, sending...
I20260808 01:49:49.823706 4786 ts_manager.cc:194] Registered new tserver with Master: 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111)
I20260808 01:49:49.825879 4786 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:51325
W20260808 01:49:50.131929 4987 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:50.132512 4987 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:50.133116 4987 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:50.165906 4987 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:50.166505 4987 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260808 01:49:50.166718 4987 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260808 01:49:50.167270 4987 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.66
I20260808 01:49:50.204357 4987 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
--webserver_interface=127.29.180.66
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.29.180.126:32885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:50.205978 4987 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:50.207922 4987 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:50.221328 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:50.830524 4983 heartbeater.cc:499] Master 127.29.180.126:32885 was elected leader, sending a full tablet report...
W20260808 01:49:51.627401 4992 debug-util.cc:398] Leaking SignalData structure 0x7b08000372c0 after lost signal to thread 4987
W20260808 01:49:51.837194 4992 kernel_stack_watchdog.cc:198] Thread 4987 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:49:50.222688 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:51.839097 4987 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.617s user 0.000s sys 0.002s
W20260808 01:49:51.839511 4987 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.618s user 0.000s sys 0.002s
W20260808 01:49:51.843979 4995 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1615 milliseconds
W20260808 01:49:51.845574 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:51.845674 4987 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:51.847081 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:51.849243 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:51.850715 4987 hybrid_clock.cc:648] HybridClock initialized: now 1786153791850673 us; error 39 us; skew 500 ppm
I20260808 01:49:51.851646 4987 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:51.858150 4987 webserver.cc:533] Webserver started at http://127.29.180.66:45401/ using document root <none> and password file <none>
I20260808 01:49:51.859413 4987 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:51.859660 4987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:51.860229 4987 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:51.864641 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/instance:
uuid: "ec65d57b7135430eb6695032355bd356"
format_stamp: "Formatted at 2026-08-08 01:49:51 on dist-test-slave-p3t5"
I20260808 01:49:51.865892 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1/instance:
uuid: "ec65d57b7135430eb6695032355bd356"
format_stamp: "Formatted at 2026-08-08 01:49:51 on dist-test-slave-p3t5"
I20260808 01:49:51.866811 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2/instance:
uuid: "ec65d57b7135430eb6695032355bd356"
format_stamp: "Formatted at 2026-08-08 01:49:51 on dist-test-slave-p3t5"
I20260808 01:49:51.867681 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance:
uuid: "ec65d57b7135430eb6695032355bd356"
format_stamp: "Formatted at 2026-08-08 01:49:51 on dist-test-slave-p3t5"
I20260808 01:49:51.883580 4987 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.008s sys 0.009s
I20260808 01:49:51.891142 5005 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:51.891856 5005 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:51.892596 5005 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:51.894025 4987 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20260808 01:49:51.894387 4987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
uuid: "ec65d57b7135430eb6695032355bd356"
format_stamp: "Formatted at 2026-08-08 01:49:51 on dist-test-slave-p3t5"
I20260808 01:49:51.894814 4987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:51.964087 4987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:51.965770 4987 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:51.966255 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:51.969017 4987 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:51.973598 4987 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:51.973953 4987 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 01:49:51.974213 4987 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:51.974398 4987 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:52.115439 4987 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.66:43479
I20260808 01:49:52.115549 5120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.66:43479 every 8 connection(s)
I20260808 01:49:52.118176 4987 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb
I20260808 01:49:52.125586 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 4987
I20260808 01:49:52.126086 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/wal/instance
I20260808 01:49:52.133188 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.67:0
--local_ip_for_outbound_sockets=127.29.180.67
--webserver_interface=127.29.180.67
--webserver_port=0
--tserver_master_addrs=127.29.180.126:32885
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
I20260808 01:49:52.141418 5121 heartbeater.cc:344] Connected to a master server at 127.29.180.126:32885
I20260808 01:49:52.141884 5121 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:52.142976 5121 heartbeater.cc:507] Master 127.29.180.126:32885 requested a full tablet report, sending...
I20260808 01:49:52.145682 4786 ts_manager.cc:194] Registered new tserver with Master: ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:52.147608 4786 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.66:40103
W20260808 01:49:52.452702 5126 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:49:52.453207 5126 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:49:52.453776 5126 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:49:52.486905 5126 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:49:52.487460 5126 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260808 01:49:52.487632 5126 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260808 01:49:52.488092 5126 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.67
I20260808 01:49:52.524861 5126 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
--webserver_interface=127.29.180.67
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.29.180.126:32885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:49:52.526418 5126 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:49:52.528908 5126 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:52.547381 5132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:53.152177 5121 heartbeater.cc:499] Master 127.29.180.126:32885 was elected leader, sending a full tablet report...
W20260808 01:49:52.552206 5135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:52.549288 5133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:53.801851 5134 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1249 milliseconds
I20260808 01:49:53.802130 5126 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260808 01:49:53.803471 5126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:49:53.805805 5126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:49:53.807190 5126 hybrid_clock.cc:648] HybridClock initialized: now 1786153793807133 us; error 75 us; skew 500 ppm
I20260808 01:49:53.808019 5126 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:53.814584 5126 webserver.cc:533] Webserver started at http://127.29.180.67:39785/ using document root <none> and password file <none>
I20260808 01:49:53.815850 5126 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:53.816094 5126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:53.816654 5126 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:53.821194 5126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/instance:
uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
format_stamp: "Formatted at 2026-08-08 01:49:53 on dist-test-slave-p3t5"
I20260808 01:49:53.822906 5126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1/instance:
uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
format_stamp: "Formatted at 2026-08-08 01:49:53 on dist-test-slave-p3t5"
I20260808 01:49:53.824262 5126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2/instance:
uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
format_stamp: "Formatted at 2026-08-08 01:49:53 on dist-test-slave-p3t5"
I20260808 01:49:53.825564 5126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance:
uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
format_stamp: "Formatted at 2026-08-08 01:49:53 on dist-test-slave-p3t5"
I20260808 01:49:53.841790 5126 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.007s sys 0.008s
I20260808 01:49:53.850292 5144 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:53.851131 5144 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:53.851787 5144 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:53.853228 5126 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260808 01:49:53.853546 5126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
format_stamp: "Formatted at 2026-08-08 01:49:53 on dist-test-slave-p3t5"
I20260808 01:49:53.853993 5126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:53.921736 5126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:49:53.923285 5126 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:49:53.923707 5126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:53.926506 5126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:53.931103 5126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:53.931301 5126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:53.931502 5126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:53.931632 5126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:49:54.076035 5126 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.67:46093
I20260808 01:49:54.076167 5259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.67:46093 every 8 connection(s)
I20260808 01:49:54.078722 5126 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb
I20260808 01:49:54.082666 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 5126
I20260808 01:49:54.083242 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/wal/instance
I20260808 01:49:54.102048 5260 heartbeater.cc:344] Connected to a master server at 127.29.180.126:32885
I20260808 01:49:54.102556 5260 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:54.103933 5260 heartbeater.cc:507] Master 127.29.180.126:32885 requested a full tablet report, sending...
I20260808 01:49:54.106441 4786 ts_manager.cc:194] Registered new tserver with Master: 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.107743 4786 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.67:40529
I20260808 01:49:54.120545 30417 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260808 01:49:54.200142 4786 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38848:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 10
seed: 0
}
}
W20260808 01:49:54.203361 4786 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:49:54.294916 4911 tablet_service.cc:1511] Processing CreateTablet for tablet 5843450fd1f54792899cc7190c94821f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.298691 4914 tablet_service.cc:1511] Processing CreateTablet for tablet 1fff9439a0bd4286827363f69521c44f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.297392 4916 tablet_service.cc:1511] Processing CreateTablet for tablet 2d7aa70246b445149411741e595f27c2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.296842 4912 tablet_service.cc:1511] Processing CreateTablet for tablet 86d58fdfa95b45f68c0648a585924d05 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.304348 4910 tablet_service.cc:1511] Processing CreateTablet for tablet 8d367581a54c433ea4f227bd4f479ed2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.308429 4909 tablet_service.cc:1511] Processing CreateTablet for tablet 2a97be89f618431b90b49c97b2c9f001 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.306110 4913 tablet_service.cc:1511] Processing CreateTablet for tablet 24ad2932d9a1489b96a63bc762f6a23f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.312835 4908 tablet_service.cc:1511] Processing CreateTablet for tablet 0c44eefb43024e9290fdba8647e745fe (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.316141 5052 tablet_service.cc:1511] Processing CreateTablet for tablet 1fff9439a0bd4286827363f69521c44f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.317227 5050 tablet_service.cc:1511] Processing CreateTablet for tablet 86d58fdfa95b45f68c0648a585924d05 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.318920 4915 tablet_service.cc:1511] Processing CreateTablet for tablet 9ee6f0c29217471e93394d4b9814144e (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.320072 5049 tablet_service.cc:1511] Processing CreateTablet for tablet 5843450fd1f54792899cc7190c94821f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.320847 5054 tablet_service.cc:1511] Processing CreateTablet for tablet 2d7aa70246b445149411741e595f27c2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.322580 5053 tablet_service.cc:1511] Processing CreateTablet for tablet 9ee6f0c29217471e93394d4b9814144e (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.325838 5051 tablet_service.cc:1511] Processing CreateTablet for tablet 24ad2932d9a1489b96a63bc762f6a23f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.329088 4907 tablet_service.cc:1511] Processing CreateTablet for tablet 213cfcc36a754065ab9203bf5a8f6f65 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.338990 5188 tablet_service.cc:1511] Processing CreateTablet for tablet 5843450fd1f54792899cc7190c94821f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.340503 5191 tablet_service.cc:1511] Processing CreateTablet for tablet 1fff9439a0bd4286827363f69521c44f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.340862 5190 tablet_service.cc:1511] Processing CreateTablet for tablet 24ad2932d9a1489b96a63bc762f6a23f (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.339437 5189 tablet_service.cc:1511] Processing CreateTablet for tablet 86d58fdfa95b45f68c0648a585924d05 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.341564 5184 tablet_service.cc:1511] Processing CreateTablet for tablet 213cfcc36a754065ab9203bf5a8f6f65 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.343312 5185 tablet_service.cc:1511] Processing CreateTablet for tablet 0c44eefb43024e9290fdba8647e745fe (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.345048 5187 tablet_service.cc:1511] Processing CreateTablet for tablet 8d367581a54c433ea4f227bd4f479ed2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.346642 5186 tablet_service.cc:1511] Processing CreateTablet for tablet 2a97be89f618431b90b49c97b2c9f001 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.350697 5192 tablet_service.cc:1511] Processing CreateTablet for tablet 9ee6f0c29217471e93394d4b9814144e (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.333133 5048 tablet_service.cc:1511] Processing CreateTablet for tablet 8d367581a54c433ea4f227bd4f479ed2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.359298 5193 tablet_service.cc:1511] Processing CreateTablet for tablet 2d7aa70246b445149411741e595f27c2 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.363559 5045 tablet_service.cc:1511] Processing CreateTablet for tablet 213cfcc36a754065ab9203bf5a8f6f65 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.361521 5046 tablet_service.cc:1511] Processing CreateTablet for tablet 0c44eefb43024e9290fdba8647e745fe (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.353376 5047 tablet_service.cc:1511] Processing CreateTablet for tablet 2a97be89f618431b90b49c97b2c9f001 (DEFAULT_TABLE table=test-workload [id=dba8af4345154a86bdfc051c4948fd47]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260808 01:49:54.410614 5285 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.419231 5286 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.456074 5285 tablet_bootstrap.cc:654] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.469141 5285 log.cc:826] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:54.469247 5288 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.475371 5289 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.481498 5291 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.484869 5290 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.494735 5285 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.498793 5285 ts_tablet_manager.cc:1403] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.008s
I20260808 01:49:54.501089 5289 tablet_bootstrap.cc:654] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.496038 5286 tablet_bootstrap.cc:654] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.510676 5283 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.512964 5288 tablet_bootstrap.cc:654] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.521600 5288 log.cc:826] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:54.523360 5291 tablet_bootstrap.cc:654] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.532225 5286 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.532176 5287 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.532781 5286 ts_tablet_manager.cc:1403] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.005s
I20260808 01:49:54.542582 5283 tablet_bootstrap.cc:654] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.545186 5292 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.557653 5288 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.558429 5288 ts_tablet_manager.cc:1403] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.090s user 0.016s sys 0.006s
I20260808 01:49:54.577137 5291 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.577606 5291 ts_tablet_manager.cc:1403] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.097s user 0.039s sys 0.020s
I20260808 01:49:54.582746 5289 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.583279 5289 ts_tablet_manager.cc:1403] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.006s
I20260808 01:49:54.583669 5283 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.584100 5283 ts_tablet_manager.cc:1403] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.005s
I20260808 01:49:54.584712 5287 tablet_bootstrap.cc:654] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.584422 5285 raft_consensus.cc:359] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.585450 5285 raft_consensus.cc:385] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.584436 5286 raft_consensus.cc:359] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.585829 5285 raft_consensus.cc:740] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.586053 5286 raft_consensus.cc:385] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.586484 5286 raft_consensus.cc:740] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.589187 5286 consensus_queue.cc:260] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.590054 5292 tablet_bootstrap.cc:654] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.587067 5285 consensus_queue.cc:260] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.587039 5283 raft_consensus.cc:359] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.592144 5283 raft_consensus.cc:385] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.592463 5283 raft_consensus.cc:740] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.593340 5283 consensus_queue.cc:260] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.593968 5292 log.cc:826] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Log is configured to *not* fsync() on all Append() calls
I20260808 01:49:54.600914 5285 ts_tablet_manager.cc:1434] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.099s user 0.034s sys 0.008s
I20260808 01:49:54.602330 5285 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.603569 5286 ts_tablet_manager.cc:1434] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.070s user 0.011s sys 0.000s
I20260808 01:49:54.604562 5286 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.605808 5291 raft_consensus.cc:359] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.606868 5291 raft_consensus.cc:385] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.607195 5291 raft_consensus.cc:740] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.608616 5288 raft_consensus.cc:359] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.610219 5289 raft_consensus.cc:359] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.610971 5289 raft_consensus.cc:385] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.610401 5291 consensus_queue.cc:260] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.611265 5289 raft_consensus.cc:740] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.611459 5286 tablet_bootstrap.cc:654] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.609543 5288 raft_consensus.cc:385] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.612057 5289 consensus_queue.cc:260] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.612618 5288 raft_consensus.cc:740] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.613394 5288 consensus_queue.cc:260] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.616928 5283 ts_tablet_manager.cc:1434] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.033s user 0.008s sys 0.004s
I20260808 01:49:54.617970 5283 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.623426 5290 tablet_bootstrap.cc:654] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.624745 5283 tablet_bootstrap.cc:654] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.625036 5286 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.625577 5286 ts_tablet_manager.cc:1403] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.010s
I20260808 01:49:54.628237 5286 raft_consensus.cc:359] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.628911 5286 raft_consensus.cc:385] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.629233 5286 raft_consensus.cc:740] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.630540 5286 consensus_queue.cc:260] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.633589 5286 ts_tablet_manager.cc:1434] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260808 01:49:54.634811 5286 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.637842 5285 tablet_bootstrap.cc:654] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.642525 5286 tablet_bootstrap.cc:654] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.650532 5289 ts_tablet_manager.cc:1434] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.066s user 0.012s sys 0.000s
I20260808 01:49:54.651961 5289 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.664682 5288 ts_tablet_manager.cc:1434] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.106s user 0.026s sys 0.008s
I20260808 01:49:54.666177 5288 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.671093 5286 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.671593 5286 ts_tablet_manager.cc:1403] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.011s
I20260808 01:49:54.674017 5291 ts_tablet_manager.cc:1434] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.095s user 0.015s sys 0.023s
I20260808 01:49:54.674979 5291 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.674814 5286 raft_consensus.cc:359] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.675544 5286 raft_consensus.cc:385] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.675839 5286 raft_consensus.cc:740] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.677520 5286 consensus_queue.cc:260] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.679574 5289 tablet_bootstrap.cc:654] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.679973 5286 ts_tablet_manager.cc:1434] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:49:54.680910 5286 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.684677 5291 tablet_bootstrap.cc:654] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.687706 5286 tablet_bootstrap.cc:654] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.688576 5288 tablet_bootstrap.cc:654] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.701092 5301 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.701718 5301 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.708166 5301 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.710474 5289 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.711158 5289 ts_tablet_manager.cc:1403] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.004s
I20260808 01:49:54.708827 5292 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.712817 5292 ts_tablet_manager.cc:1403] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.168s user 0.019s sys 0.000s
I20260808 01:49:54.719565 5289 raft_consensus.cc:359] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.721211 5285 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.721779 5285 ts_tablet_manager.cc:1403] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.120s user 0.006s sys 0.018s
I20260808 01:49:54.722400 5289 raft_consensus.cc:385] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.723577 5289 raft_consensus.cc:740] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.724941 5285 raft_consensus.cc:359] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.727063 5290 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.727501 5290 ts_tablet_manager.cc:1403] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.243s user 0.014s sys 0.003s
I20260808 01:49:54.725216 5289 consensus_queue.cc:260] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.731231 5289 ts_tablet_manager.cc:1434] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260808 01:49:54.732139 5289 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.733008 5288 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.733495 5288 ts_tablet_manager.cc:1403] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.004s
I20260808 01:49:54.736443 5288 raft_consensus.cc:359] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.737182 5288 raft_consensus.cc:385] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.737479 5288 raft_consensus.cc:740] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.740665 5288 consensus_queue.cc:260] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.744102 5287 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.744539 5287 ts_tablet_manager.cc:1403] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.215s user 0.015s sys 0.004s
I20260808 01:49:54.745175 5289 tablet_bootstrap.cc:654] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.745662 5288 ts_tablet_manager.cc:1434] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260808 01:49:54.754626 5291 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.755123 5291 ts_tablet_manager.cc:1403] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.008s
I20260808 01:49:54.757994 5291 raft_consensus.cc:359] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.758710 5291 raft_consensus.cc:385] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.759006 5291 raft_consensus.cc:740] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.759709 5291 consensus_queue.cc:260] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.760558 5286 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.761157 5286 ts_tablet_manager.cc:1403] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.003s
I20260808 01:49:54.764199 5286 raft_consensus.cc:359] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.764741 5291 ts_tablet_manager.cc:1434] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20260808 01:49:54.765735 5291 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.754657 5288 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.773416 5288 tablet_bootstrap.cc:654] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.773862 5286 raft_consensus.cc:385] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.774271 5286 raft_consensus.cc:740] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.739950 5285 raft_consensus.cc:385] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.774782 5285 raft_consensus.cc:740] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.775682 5285 consensus_queue.cc:260] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.779160 5301 raft_consensus.cc:493] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.779592 5301 raft_consensus.cc:515] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.783679 5301 leader_election.cc:290] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.783842 5283 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.784301 5283 ts_tablet_manager.cc:1403] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.167s user 0.025s sys 0.048s
I20260808 01:49:54.785399 5285 ts_tablet_manager.cc:1434] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.063s user 0.002s sys 0.005s
I20260808 01:49:54.791785 5285 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:49:54.787642 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.794274 5283 raft_consensus.cc:359] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.794931 5283 raft_consensus.cc:385] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.788524 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.795188 5283 raft_consensus.cc:740] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.786255 5286 consensus_queue.cc:260] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:49:54.796255 4984 tablet.cc:2406] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:54.795889 5283 consensus_queue.cc:260] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.797657 5286 ts_tablet_manager.cc:1434] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260808 01:49:54.798139 5283 ts_tablet_manager.cc:1434] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.014s user 0.001s sys 0.007s
I20260808 01:49:54.798655 5286 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
W20260808 01:49:54.799077 4869 leader_election.cc:343] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:54.799798 4869 leader_election.cc:343] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.804939 5286 tablet_bootstrap.cc:654] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.807194 5299 raft_consensus.cc:493] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.807677 5299 raft_consensus.cc:515] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.809759 5299 leader_election.cc:290] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.811048 5299 raft_consensus.cc:493] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.811432 5299 raft_consensus.cc:515] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.811959 5290 raft_consensus.cc:359] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.814956 5299 leader_election.cc:290] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.815109 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:54.790412 5289 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.816128 5289 ts_tablet_manager.cc:1403] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.010s
I20260808 01:49:54.817157 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
W20260808 01:49:54.818970 4869 leader_election.cc:343] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.819047 5289 raft_consensus.cc:359] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.819712 5289 raft_consensus.cc:385] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.819988 5289 raft_consensus.cc:740] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
W20260808 01:49:54.819762 4869 leader_election.cc:343] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.821369 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.821758 5290 raft_consensus.cc:385] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.814338 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.822081 5290 raft_consensus.cc:740] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.793594 5075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:54.823103 5075 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:54.823017 5290 consensus_queue.cc:260] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.794744 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:54.800485 5291 tablet_bootstrap.cc:654] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.820793 5292 raft_consensus.cc:359] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.825109 5292 raft_consensus.cc:385] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.825392 5292 raft_consensus.cc:740] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.826283 5292 consensus_queue.cc:260] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.828711 5287 raft_consensus.cc:359] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.829394 5287 raft_consensus.cc:385] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.829486 5285 tablet_bootstrap.cc:654] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.829671 5287 raft_consensus.cc:740] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.830375 5287 consensus_queue.cc:260] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.835193 5289 consensus_queue.cc:260] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.837436 5289 ts_tablet_manager.cc:1434] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.021s user 0.001s sys 0.004s
I20260808 01:49:54.839107 5289 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356: Bootstrap starting.
I20260808 01:49:54.840503 4869 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:54.836418 5288 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.842415 5288 ts_tablet_manager.cc:1403] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.011s
I20260808 01:49:54.845531 5288 raft_consensus.cc:359] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.846330 5288 raft_consensus.cc:385] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.846657 5288 raft_consensus.cc:740] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.843675 5299 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:54.848742 5292 ts_tablet_manager.cc:1434] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.136s user 0.023s sys 0.000s
I20260808 01:49:54.848852 5299 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:54.850699 5287 ts_tablet_manager.cc:1434] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.106s user 0.012s sys 0.006s
W20260808 01:49:54.851444 4869 leader_election.cc:343] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.852049 4869 leader_election.cc:304] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356
I20260808 01:49:54.849243 5299 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:54.852978 5291 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.853533 5291 ts_tablet_manager.cc:1403] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.003s
I20260808 01:49:54.854871 5301 raft_consensus.cc:2749] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:54.855448 5288 consensus_queue.cc:260] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.857239 5291 raft_consensus.cc:359] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.858139 5291 raft_consensus.cc:385] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.858443 5291 raft_consensus.cc:740] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.859536 5260 heartbeater.cc:499] Master 127.29.180.126:32885 was elected leader, sending a full tablet report...
I20260808 01:49:54.869941 5291 consensus_queue.cc:260] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.872224 5291 ts_tablet_manager.cc:1434] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.018s user 0.003s sys 0.001s
I20260808 01:49:54.872319 5286 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.872793 5286 ts_tablet_manager.cc:1403] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.012s
W20260808 01:49:54.873078 4869 leader_election.cc:343] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.874248 5289 tablet_bootstrap.cc:654] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.875499 5288 ts_tablet_manager.cc:1434] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.033s user 0.007s sys 0.001s
I20260808 01:49:54.875994 5290 ts_tablet_manager.cc:1434] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.148s user 0.017s sys 0.001s
W20260808 01:49:54.876276 5122 tablet.cc:2406] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:54.877074 5290 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.880897 4869 leader_election.cc:304] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356
I20260808 01:49:54.882193 5301 raft_consensus.cc:2749] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:54.880532 5286 raft_consensus.cc:359] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.883668 5286 raft_consensus.cc:385] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.884017 5286 raft_consensus.cc:740] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
W20260808 01:49:54.883271 4869 leader_election.cc:343] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.884449 5299 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.884868 4869 leader_election.cc:304] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356
I20260808 01:49:54.885742 5301 raft_consensus.cc:2749] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:54.886392 5286 consensus_queue.cc:260] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.889670 5287 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.889933 5285 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: No bootstrap required, opened a new log
I20260808 01:49:54.890820 5285 ts_tablet_manager.cc:1403] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.013s
I20260808 01:49:54.892902 5301 raft_consensus.cc:493] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.893369 5301 raft_consensus.cc:515] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.895431 5301 leader_election.cc:290] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.897006 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:54.897321 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.897657 5074 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:54.898095 5212 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:54.899613 4869 leader_election.cc:304] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:54.900775 5292 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.901224 5301 raft_consensus.cc:2804] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:54.901573 5301 raft_consensus.cc:493] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:54.901947 5301 raft_consensus.cc:3060] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:54.902175 5286 ts_tablet_manager.cc:1434] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.029s user 0.001s sys 0.007s
I20260808 01:49:54.903795 5285 raft_consensus.cc:359] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.904489 5285 raft_consensus.cc:385] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.904824 5285 raft_consensus.cc:740] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.909109 5301 raft_consensus.cc:515] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.911099 5301 leader_election.cc:290] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.912359 5289 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356: No bootstrap required, opened a new log
I20260808 01:49:54.912884 5289 ts_tablet_manager.cc:1403] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.015s
I20260808 01:49:54.912678 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:54.912796 5285 consensus_queue.cc:260] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.913385 5074 raft_consensus.cc:3060] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:54.913630 5299 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.916059 5289 raft_consensus.cc:359] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.916838 5289 raft_consensus.cc:385] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.917192 5289 raft_consensus.cc:740] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65d57b7135430eb6695032355bd356, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.919967 5290 tablet_bootstrap.cc:654] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.920931 5074 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:54.920888 5289 consensus_queue.cc:260] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.921706 5285 ts_tablet_manager.cc:1434] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260808 01:49:54.922892 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:54.923511 5212 raft_consensus.cc:3060] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:54.929184 5292 tablet_bootstrap.cc:654] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.930063 5289 ts_tablet_manager.cc:1434] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20260808 01:49:54.935148 5287 tablet_bootstrap.cc:654] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:54.939114 5212 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:54.943603 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:54.945703 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:54.946770 4869 leader_election.cc:304] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:54.947633 5074 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:54.954376 5301 raft_consensus.cc:2804] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:54.955020 5301 raft_consensus.cc:697] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:54.956203 5301 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.963675 5305 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:54.963773 4869 leader_election.cc:343] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.964228 5305 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.970291 5287 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.972985 5287 ts_tablet_manager.cc:1403] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.008s
I20260808 01:49:54.973407 5290 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.972414 5292 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:54.973896 5290 ts_tablet_manager.cc:1403] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.007s
I20260808 01:49:54.974043 5292 ts_tablet_manager.cc:1403] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.017s
I20260808 01:49:54.976619 5290 raft_consensus.cc:359] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.977337 5290 raft_consensus.cc:385] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.977669 5290 raft_consensus.cc:740] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.977679 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:54.978396 5290 consensus_queue.cc:260] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.980495 5290 ts_tablet_manager.cc:1434] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260808 01:49:54.980530 5287 raft_consensus.cc:359] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.981940 5287 raft_consensus.cc:385] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.982293 5287 raft_consensus.cc:740] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:54.983106 5287 consensus_queue.cc:260] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.981686 5290 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.977084 5292 raft_consensus.cc:359] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.985548 5074 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:54.985823 5292 raft_consensus.cc:385] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:54.986822 5301 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.987283 5301 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.989360 5301 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.992223 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:54.992862 5287 ts_tablet_manager.cc:1434] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.019s user 0.002s sys 0.005s
I20260808 01:49:54.993549 5321 raft_consensus.cc:493] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.994127 5287 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:54.994084 5321 raft_consensus.cc:515] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:49:54.998517 4869 leader_election.cc:343] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:54.999191 5301 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:54.999711 5301 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:54.997370 5321 leader_election.cc:290] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.004209 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.005162 5305 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:49:55.005975 4869 leader_election.cc:343] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:55.006680 5301 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:54.986236 5292 raft_consensus.cc:740] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:55.008116 5292 consensus_queue.cc:260] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.011270 5292 ts_tablet_manager.cc:1434] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.037s user 0.004s sys 0.002s
I20260808 01:49:55.012557 5292 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:55.016911 5290 tablet_bootstrap.cc:654] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:55.021452 5292 tablet_bootstrap.cc:654] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:55.033725 5287 tablet_bootstrap.cc:654] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:55.051003 5290 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:55.053830 5292 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:55.054544 5292 ts_tablet_manager.cc:1403] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.013s
I20260808 01:49:55.057699 5292 raft_consensus.cc:359] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.058241 5301 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.059018 5292 raft_consensus.cc:385] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:55.058712 5301 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.059444 5292 raft_consensus.cc:740] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:55.060253 5292 consensus_queue.cc:260] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.060808 5301 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.058717 5290 ts_tablet_manager.cc:1403] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.007s
I20260808 01:49:55.064078 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.064677 5212 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.064895 5290 raft_consensus.cc:359] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.065539 5290 raft_consensus.cc:385] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:55.065835 5290 raft_consensus.cc:740] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:55.066192 4869 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:55.066545 5290 consensus_queue.cc:260] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.067137 5301 raft_consensus.cc:2804] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.067718 5301 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.068123 5301 raft_consensus.cc:3060] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.070844 5292 ts_tablet_manager.cc:1434] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260808 01:49:55.070966 5287 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:55.071583 5287 ts_tablet_manager.cc:1403] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.012s
I20260808 01:49:55.071796 5292 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: Bootstrap starting.
I20260808 01:49:55.078302 5287 raft_consensus.cc:359] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.079072 5287 raft_consensus.cc:385] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:55.079368 5287 raft_consensus.cc:740] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:55.079802 5290 ts_tablet_manager.cc:1434] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260808 01:49:55.078338 5292 tablet_bootstrap.cc:654] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:55.080055 5287 consensus_queue.cc:260] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:49:55.084586 5261 tablet.cc:2406] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:49:55.087538 5287 ts_tablet_manager.cc:1434] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260808 01:49:55.101830 5299 raft_consensus.cc:493] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.102401 5299 raft_consensus.cc:515] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.104475 5299 leader_election.cc:290] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.105075 5292 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: No bootstrap required, opened a new log
I20260808 01:49:55.105566 5292 ts_tablet_manager.cc:1403] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.011s
I20260808 01:49:55.106508 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.107136 5212 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.108475 5292 raft_consensus.cc:359] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.109192 5292 raft_consensus.cc:385] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:55.108872 4869 leader_election.cc:304] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:55.109478 5292 raft_consensus.cc:740] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Initialized, Role: FOLLOWER
I20260808 01:49:55.110357 5299 raft_consensus.cc:2804] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.110246 5292 consensus_queue.cc:260] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.110857 5299 raft_consensus.cc:493] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.111421 5299 raft_consensus.cc:3060] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.112820 5292 ts_tablet_manager.cc:1434] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 01:49:55.118558 5301 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.120764 5321 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.121250 5321 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.123226 5321 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.120625 5301 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.126161 5305 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.126621 5305 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.128696 5305 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.128408 5299 raft_consensus.cc:515] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.129333 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.130036 5212 raft_consensus.cc:3060] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.130770 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.131368 5074 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.131601 4869 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:55.132859 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.133436 5074 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.134359 5075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.134943 5075 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.135308 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.135834 5072 raft_consensus.cc:3060] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.135733 4869 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:55.136626 5300 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.136893 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.137452 5074 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.137792 5301 raft_consensus.cc:2804] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.138129 5301 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.138456 5301 raft_consensus.cc:3060] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.138847 5300 raft_consensus.cc:697] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:55.139875 4869 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:55.139739 5300 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.141042 5343 raft_consensus.cc:2804] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.141398 5343 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.141825 5343 raft_consensus.cc:3060] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.143589 5072 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.144490 5299 leader_election.cc:290] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.144989 5299 raft_consensus.cc:493] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.145260 4869 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.145390 5299 raft_consensus.cc:515] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.147316 5299 leader_election.cc:290] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.147933 5299 raft_consensus.cc:2804] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.148312 5299 raft_consensus.cc:697] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:55.149092 5299 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.154912 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.155566 5072 raft_consensus.cc:3060] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.158456 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.158617 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.159030 5213 raft_consensus.cc:3060] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.159518 4936 raft_consensus.cc:2468] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 0.
I20260808 01:49:55.159670 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.160256 5074 raft_consensus.cc:2468] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 0.
I20260808 01:49:55.161022 4935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.162065 4869 leader_election.cc:304] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.162499 5147 leader_election.cc:304] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:55.163154 5344 raft_consensus.cc:2804] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.163344 5321 raft_consensus.cc:2804] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.163508 5344 raft_consensus.cc:493] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.163697 5321 raft_consensus.cc:493] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.163813 5344 raft_consensus.cc:3060] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.164146 5305 raft_consensus.cc:493] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.164580 5305 raft_consensus.cc:515] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.166687 5305 leader_election.cc:290] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.164042 5321 raft_consensus.cc:3060] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.171387 5072 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.171135 5301 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.171427 5211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.172580 4935 raft_consensus.cc:2393] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.172922 4869 leader_election.cc:304] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.174265 5300 raft_consensus.cc:2804] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.177033 5212 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.178161 5301 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.180052 5319 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.180142 5321 raft_consensus.cc:515] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.180575 5319 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.182809 5319 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.183549 5211 raft_consensus.cc:2393] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01adc8a4bde24946b6bf7e18e876e977 in current term 1: Already voted for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in this term.
I20260808 01:49:55.184151 5213 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.184955 5321 leader_election.cc:290] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.186506 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:49:55.187489 4935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.191958 5343 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.193351 4935 raft_consensus.cc:2393] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.194247 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.194952 5074 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 0.
I20260808 01:49:55.196265 5075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.197026 5075 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 0.
I20260808 01:49:55.198249 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.198928 5073 raft_consensus.cc:3060] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.202389 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.203773 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.204241 5146 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:49:55.204393 5213 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.205557 5146 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:49:55.205946 5007 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.208154 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.208178 5075 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.208580 5343 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.208760 5075 raft_consensus.cc:3060] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.208748 5212 raft_consensus.cc:2468] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.209931 5007 leader_election.cc:304] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.212844 5210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.213800 5319 raft_consensus.cc:2804] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.203140 5300 raft_consensus.cc:697] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:55.214140 5319 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.214454 5319 raft_consensus.cc:3060] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.214634 5300 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.215773 5349 raft_consensus.cc:2804] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.216128 5349 raft_consensus.cc:493] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.216480 5349 raft_consensus.cc:3060] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.213474 5210 raft_consensus.cc:3060] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.217842 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.218756 5211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.200539 5344 raft_consensus.cc:515] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.221189 5211 raft_consensus.cc:2393] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec65d57b7135430eb6695032355bd356 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.221593 4935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.222522 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.223081 5074 raft_consensus.cc:3060] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.223531 4933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.224385 4933 raft_consensus.cc:2468] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.225780 4934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.226655 4934 raft_consensus.cc:2393] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec65d57b7135430eb6695032355bd356 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.229053 5210 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.229915 5074 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.229992 5008 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:55.230943 5321 raft_consensus.cc:2764] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:49:55.231707 4936 raft_consensus.cc:2393] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.231823 4869 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9; no voters:
I20260808 01:49:55.232836 5343 raft_consensus.cc:2804] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.233336 5343 raft_consensus.cc:697] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:55.234308 5343 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.234755 5319 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.236193 5213 raft_consensus.cc:2393] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 01adc8a4bde24946b6bf7e18e876e977 in current term 1: Already voted for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in this term.
I20260808 01:49:55.235982 5344 leader_election.cc:290] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Requested vote from peers ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.239318 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.240016 5213 raft_consensus.cc:2393] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 01adc8a4bde24946b6bf7e18e876e977 in current term 1: Already voted for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in this term.
I20260808 01:49:55.222510 5073 raft_consensus.cc:2468] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:49:55.242623 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:49:55.243597 4936 raft_consensus.cc:2393] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.243934 5146 leader_election.cc:304] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:49:55.244769 5321 raft_consensus.cc:2804] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.243935 5319 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.242882 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.246009 5321 raft_consensus.cc:697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:49:55.246809 5146 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9; no voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356
I20260808 01:49:55.247012 5321 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.247661 5351 raft_consensus.cc:2749] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:55.247958 5319 raft_consensus.cc:2749] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:55.249778 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5843450fd1f54792899cc7190c94821f" candidate_uuid: "01adc8a4bde24946b6bf7e18e876e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:49:55.250196 5075 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01adc8a4bde24946b6bf7e18e876e977 in term 1.
I20260808 01:49:55.250576 5073 raft_consensus.cc:2393] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 01adc8a4bde24946b6bf7e18e876e977 in current term 1: Already voted for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in this term.
I20260808 01:49:55.250773 5305 raft_consensus.cc:2764] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:49:55.253904 4869 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:49:55.255075 5347 raft_consensus.cc:2804] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.255604 5347 raft_consensus.cc:697] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 LEADER]: Becoming Leader. State: Replica: 01adc8a4bde24946b6bf7e18e876e977, State: Running, Role: LEADER
I20260808 01:49:55.256598 5347 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.256924 5349 raft_consensus.cc:515] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.262394 4869 leader_election.cc:304] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977; no voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356
I20260808 01:49:55.264717 5305 raft_consensus.cc:493] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.265462 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:49:55.265362 5305 raft_consensus.cc:515] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.266064 4936 raft_consensus.cc:3060] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.266340 5349 leader_election.cc:290] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.268400 5305 leader_election.cc:290] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.268743 4935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.269148 5343 raft_consensus.cc:2749] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:55.270813 5210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.271443 5210 raft_consensus.cc:2468] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.273020 5007 leader_election.cc:304] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.269380 4935 raft_consensus.cc:2468] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.264701 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.277817 5305 raft_consensus.cc:2804] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.278735 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.279826 5305 raft_consensus.cc:493] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.281745 5305 raft_consensus.cc:3060] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.284265 4936 raft_consensus.cc:2468] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.285602 5008 leader_election.cc:304] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.280004 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:55.286427 5349 raft_consensus.cc:2804] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.286922 5349 raft_consensus.cc:697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Becoming Leader. State: Replica: ec65d57b7135430eb6695032355bd356, State: Running, Role: LEADER
I20260808 01:49:55.286545 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.287897 5349 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.289049 5213 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a97be89f618431b90b49c97b2c9f001" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.289669 5213 raft_consensus.cc:3060] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.308429 5305 raft_consensus.cc:515] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.311111 5305 leader_election.cc:290] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.313530 5210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.314201 5210 raft_consensus.cc:3060] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.319254 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ee6f0c29217471e93394d4b9814144e" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:49:55.319859 4936 raft_consensus.cc:3060] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.323287 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.326651 4936 raft_consensus.cc:2468] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.327994 5008 leader_election.cc:304] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.328632 5349 raft_consensus.cc:493] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.329720 5213 raft_consensus.cc:2468] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.326234 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65d57b7135430eb6695032355bd356 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.331682 5349 raft_consensus.cc:515] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.327733 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65). New cstate: current_term: 1 leader_uuid: "01adc8a4bde24946b6bf7e18e876e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.330649 5305 raft_consensus.cc:2804] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.334085 5305 raft_consensus.cc:697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Becoming Leader. State: Replica: ec65d57b7135430eb6695032355bd356, State: Running, Role: LEADER
I20260808 01:49:55.335173 5210 raft_consensus.cc:2468] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.334961 5305 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.339350 5349 leader_election.cc:290] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.339434 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.340065 4936 raft_consensus.cc:2468] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.341324 5008 leader_election.cc:304] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.342226 5349 raft_consensus.cc:2804] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:55.342684 5349 raft_consensus.cc:493] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:55.340227 5210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:49:55.343684 5349 raft_consensus.cc:3060] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.344353 5210 raft_consensus.cc:2468] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 0.
I20260808 01:49:55.358824 5349 raft_consensus.cc:515] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.361820 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:49:55.362428 4936 raft_consensus.cc:3060] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.367905 5349 leader_election.cc:290] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:49:55.368141 5210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "213cfcc36a754065ab9203bf5a8f6f65" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:49:55.368759 5210 raft_consensus.cc:3060] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.368973 4936 raft_consensus.cc:2468] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.370415 5008 leader_election.cc:304] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:49:55.371292 5349 raft_consensus.cc:2804] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:55.374487 5349 raft_consensus.cc:697] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Becoming Leader. State: Replica: ec65d57b7135430eb6695032355bd356, State: Running, Role: LEADER
I20260808 01:49:55.375342 5349 consensus_queue.cc:237] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.376327 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65d57b7135430eb6695032355bd356 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.392532 5210 raft_consensus.cc:2468] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:49:55.399531 4786 catalog_manager.cc:5697] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65d57b7135430eb6695032355bd356 (127.29.180.66). New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:55.412669 5321 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.413269 5321 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:49:55.414969 5321 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:49:55.416018 4936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
I20260808 01:49:55.416235 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:49:55.416980 5073 raft_consensus.cc:2393] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 1: Already voted for candidate 01adc8a4bde24946b6bf7e18e876e977 in this term.
I20260808 01:49:55.417915 5146 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9; no voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356
I20260808 01:49:55.418552 5321 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:55.423445 5321 raft_consensus.cc:2749] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:55.526962 5347 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.537765 5300 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.542541 5300 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.557384 5343 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:49:55.596186 5343 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.608726 5344 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.627895 5344 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.641481 5347 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.732250 5321 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:55.764664 5349 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.775035 5344 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.781124 5344 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:55.806001 5351 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.807572 5321 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.822952 5347 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:55.833061 5347 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:55.884593 5349 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.895697 5349 consensus_queue.cc:1048] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:49:55.896251 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 56394.8us
I20260808 01:49:55.900470 5349 consensus_queue.cc:1048] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:55.920933 5305 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:56.345606 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.000728
I20260808 01:49:56.363701 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.001678
W20260808 01:49:56.437655 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 73726.9us
I20260808 01:49:56.463201 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.001678
W20260808 01:49:56.535037 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 137204us
I20260808 01:49:56.859104 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.512s user 0.064s sys 0.029s Metrics: {"bytes_written":4711,"cfile_init":1,"compiler_manager_pool.queue_time_us":2436,"dirs.queue_time_us":37982,"dirs.run_cpu_time_us":401,"dirs.run_wall_time_us":4522,"drs_written":1,"lbm_read_time_us":412,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":3,"thread_start_us":69852,"threads_started":2}
I20260808 01:49:56.868481 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 171 bytes on disk
I20260808 01:49:56.871682 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":488,"lbm_reads_lt_1ms":4}
I20260808 01:49:56.877003 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.003577
W20260808 01:49:56.911598 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 172386us
I20260808 01:49:57.146183 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.266s user 0.063s sys 0.057s Metrics: {"bytes_written":5009,"cfile_init":1,"dirs.queue_time_us":156185,"dirs.run_cpu_time_us":351,"dirs.run_wall_time_us":1046,"drs_written":1,"lbm_read_time_us":314,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2778,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":16,"thread_start_us":158935,"threads_started":1}
I20260808 01:49:57.153232 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.003577
I20260808 01:49:57.280568 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.914s user 0.061s sys 0.030s Metrics: {"bytes_written":4781,"cfile_init":1,"compiler_manager_pool.queue_time_us":10310,"dirs.queue_time_us":32797,"dirs.run_cpu_time_us":528,"dirs.run_wall_time_us":13848,"drs_written":1,"lbm_read_time_us":371,"lbm_reads_lt_1ms":4,"lbm_write_time_us":55435,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":7,"spinlock_wait_cycles":77568,"thread_start_us":36165,"threads_started":2}
I20260808 01:49:57.323698 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.007377
W20260808 01:49:57.439708 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 62716.4us
I20260808 01:49:57.587888 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.112s user 0.058s sys 0.020s Metrics: {"bytes_written":4781,"cfile_init":1,"compiler_manager_pool.queue_time_us":61983,"dirs.queue_time_us":32285,"dirs.run_cpu_time_us":377,"dirs.run_wall_time_us":3649,"drs_written":1,"lbm_read_time_us":418,"lbm_reads_lt_1ms":4,"lbm_write_time_us":40346,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":7,"thread_start_us":98657,"threads_started":2}
I20260808 01:49:57.662108 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.007377
W20260808 01:49:57.700248 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 177358us
I20260808 01:49:57.985536 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.826s user 0.035s sys 0.017s Metrics: {"bytes_written":5450,"cfile_init":1,"dirs.queue_time_us":105910,"dirs.run_cpu_time_us":407,"dirs.run_wall_time_us":26065,"drs_written":1,"lbm_read_time_us":418,"lbm_reads_lt_1ms":4,"lbm_write_time_us":40882,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":32,"thread_start_us":107959,"threads_started":1}
I20260808 01:49:57.994557 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 417 bytes on disk
I20260808 01:49:57.997036 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":313,"lbm_reads_lt_1ms":4}
I20260808 01:49:57.969549 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.645s user 0.039s sys 0.015s Metrics: {"bytes_written":5450,"cfile_init":1,"dirs.queue_time_us":82272,"dirs.run_cpu_time_us":461,"dirs.run_wall_time_us":35201,"drs_written":1,"lbm_read_time_us":441,"lbm_reads_lt_1ms":4,"lbm_write_time_us":47145,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":32,"thread_start_us":78311,"threads_started":1}
I20260808 01:49:58.036001 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 417 bytes on disk
I20260808 01:49:58.038462 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":426,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.031976 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.367s user 0.054s sys 0.023s Metrics: {"bytes_written":5450,"cfile_init":1,"dirs.queue_time_us":106682,"dirs.run_cpu_time_us":391,"dirs.run_wall_time_us":45661,"drs_written":1,"lbm_read_time_us":342,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13187,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":32,"thread_start_us":115892,"threads_started":1}
I20260808 01:49:58.051179 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 417 bytes on disk
I20260808 01:49:58.053115 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 283 bytes on disk
I20260808 01:49:58.053962 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":321,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.056531 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.007377
I20260808 01:49:58.061738 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.007377
W20260808 01:49:58.089635 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 86428.2us
I20260808 01:49:58.100471 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.121474 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.007377
I20260808 01:49:58.542804 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.477s user 0.038s sys 0.014s Metrics: {"bytes_written":5827,"cfile_init":1,"dirs.queue_time_us":12113,"dirs.run_cpu_time_us":401,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":40383,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":13452,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":45}
I20260808 01:49:58.552505 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.007377
I20260808 01:49:58.657364 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.565s user 0.048s sys 0.006s Metrics: {"bytes_written":5827,"cfile_init":1,"dirs.queue_time_us":56408,"dirs.run_cpu_time_us":342,"dirs.run_wall_time_us":23107,"drs_written":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":45,"thread_start_us":56664,"threads_started":1}
I20260808 01:49:58.671401 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.007377
I20260808 01:49:58.746680 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.624s user 0.044s sys 0.015s Metrics: {"bytes_written":5827,"cfile_init":1,"dirs.queue_time_us":63718,"dirs.run_cpu_time_us":344,"dirs.run_wall_time_us":4522,"drs_written":1,"lbm_read_time_us":279,"lbm_reads_lt_1ms":4,"lbm_write_time_us":27874,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":45,"thread_start_us":65577,"threads_started":1}
I20260808 01:49:58.793881 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.237s user 0.043s sys 0.004s Metrics: {"bytes_written":5633,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1001,"drs_written":1,"lbm_read_time_us":290,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2512,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":38}
I20260808 01:49:58.802821 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.007377
I20260808 01:49:58.812623 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 548 bytes on disk
I20260808 01:49:58.818506 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":352,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.832405 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.007377
I20260808 01:49:58.847333 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.175s user 0.046s sys 0.001s Metrics: {"bytes_written":5633,"cfile_init":1,"dirs.queue_time_us":23176,"dirs.run_cpu_time_us":456,"dirs.run_wall_time_us":13018,"drs_written":1,"lbm_read_time_us":360,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2668,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":38,"thread_start_us":23533,"threads_started":1}
I20260808 01:49:58.874634 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 548 bytes on disk
I20260808 01:49:58.877121 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":359,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.885496 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.007377
W20260808 01:49:58.921586 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 53783.8us
I20260808 01:49:58.961371 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.127s user 0.037s sys 0.010s Metrics: {"bytes_written":5967,"cfile_init":1,"dirs.queue_time_us":7387,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":7646,"drs_written":1,"lbm_read_time_us":257,"lbm_reads_lt_1ms":4,"lbm_write_time_us":18553,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":49,"thread_start_us":7611,"threads_started":1}
I20260808 01:49:58.971992 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 608 bytes on disk
I20260808 01:49:58.973769 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":266,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.978897 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 499 bytes on disk
I20260808 01:49:58.980588 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4}
I20260808 01:49:58.986038 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.007377
I20260808 01:49:59.045775 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.059s user 0.047s sys 0.003s Metrics: {"bytes_written":6212,"cfile_init":1,"dirs.queue_time_us":116,"dirs.run_cpu_time_us":328,"dirs.run_wall_time_us":3704,"drs_written":1,"lbm_read_time_us":259,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":59}
I20260808 01:49:59.052861 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.007377
W20260808 01:49:59.283037 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 97760.3us
I20260808 01:49:59.413862 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.521s user 0.051s sys 0.002s Metrics: {"bytes_written":6251,"cfile_init":1,"dirs.queue_time_us":41083,"dirs.run_cpu_time_us":378,"dirs.run_wall_time_us":32055,"drs_written":1,"lbm_read_time_us":287,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2282,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":61,"thread_start_us":44270,"threads_started":1}
I20260808 01:49:59.423007 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.014976
W20260808 01:49:59.425778 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 141716us
I20260808 01:49:59.705866 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.652s user 0.056s sys 0.008s Metrics: {"bytes_written":5827,"cfile_init":1,"dirs.queue_time_us":56395,"dirs.run_cpu_time_us":417,"dirs.run_wall_time_us":5859,"drs_written":1,"lbm_read_time_us":4887,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":44026,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":45,"thread_start_us":23658,"threads_started":1}
I20260808 01:49:59.714727 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.008746
W20260808 01:49:59.817757 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 61219.2us
W20260808 01:49:59.911113 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.004s sys 0.000s
I20260808 01:49:59.935891 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 1.126s user 0.036s sys 0.024s Metrics: {"bytes_written":5967,"cfile_init":1,"dirs.queue_time_us":39233,"dirs.run_cpu_time_us":398,"dirs.run_wall_time_us":9937,"drs_written":1,"lbm_read_time_us":263,"lbm_reads_lt_1ms":4,"lbm_write_time_us":134687,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":49,"thread_start_us":85678,"threads_started":1}
I20260808 01:49:59.944404 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.014976
W20260808 01:50:00.057101 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 57864.5us
I20260808 01:50:00.033505 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.610s user 0.114s sys 0.040s Metrics: {"bytes_written":6715,"cfile_init":1,"dirs.queue_time_us":316786,"dirs.run_cpu_time_us":408,"dirs.run_wall_time_us":20492,"drs_written":1,"lbm_read_time_us":286,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":79,"thread_start_us":316471,"threads_started":1}
W20260808 01:50:00.344767 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 84557.3us
I20260808 01:50:00.360355 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 891 bytes on disk
I20260808 01:50:00.362596 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":376,"lbm_reads_lt_1ms":4}
I20260808 01:50:00.413800 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.014976
W20260808 01:50:00.460371 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 239423us
W20260808 01:50:00.553788 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 50963us
I20260808 01:50:00.670035 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.698s user 0.054s sys 0.012s Metrics: {"bytes_written":6514,"cfile_init":1,"dirs.queue_time_us":61688,"dirs.run_cpu_time_us":394,"dirs.run_wall_time_us":9919,"drs_written":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2401,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":72,"thread_start_us":71789,"threads_started":1}
I20260808 01:50:00.695529 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.861116
W20260808 01:50:00.726418 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 87658.2us
I20260808 01:50:00.839015 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.411s user 0.049s sys 0.012s Metrics: {"bytes_written":6658,"cfile_init":1,"dirs.queue_time_us":13779,"dirs.run_cpu_time_us":389,"dirs.run_wall_time_us":19112,"drs_written":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":76,"thread_start_us":48459,"threads_started":1}
I20260808 01:50:00.872953 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.157s user 0.078s sys 0.028s Metrics: {"bytes_written":6681,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1619,"cfile_cache_miss":26,"cfile_cache_miss_bytes":4867,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":105028,"dirs.run_cpu_time_us":332,"dirs.run_wall_time_us":70736,"drs_written":1,"lbm_read_time_us":13671,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":48,"lbm_write_time_us":19546,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":77,"spinlock_wait_cycles":13200256,"thread_start_us":105820,"threads_started":1}
I20260808 01:50:00.882195 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 867 bytes on disk
I20260808 01:50:00.889369 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":313,"lbm_reads_lt_1ms":4}
I20260808 01:50:00.983314 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 869 bytes on disk
I20260808 01:50:00.994475 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":415,"lbm_reads_lt_1ms":4}
I20260808 01:50:01.010540 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 695 bytes on disk
I20260808 01:50:01.038110 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 724 bytes on disk
I20260808 01:50:01.040606 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
I20260808 01:50:01.043239 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4}
W20260808 01:50:01.108691 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 53925.1us
I20260808 01:50:01.150981 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 499 bytes on disk
I20260808 01:50:01.153731 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":331,"lbm_reads_lt_1ms":4}
I20260808 01:50:01.141327 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.014976
W20260808 01:50:01.210937 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 54902.5us
I20260808 01:50:01.196851 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.014976
W20260808 01:50:01.258685 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 44094.7us
W20260808 01:50:01.826586 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 290555us
I20260808 01:50:01.878690 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.181s user 0.109s sys 0.029s Metrics: {"bytes_written":6933,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1814,"cfile_cache_miss":26,"cfile_cache_miss_bytes":5400,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":369443,"dirs.run_cpu_time_us":4158,"dirs.run_wall_time_us":38270,"drs_written":1,"lbm_read_time_us":4367,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2757,"lbm_writes_lt_1ms":23,"mutex_wait_us":294,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":88,"spinlock_wait_cycles":15232,"thread_start_us":411269,"threads_started":6}
I20260808 01:50:01.953293 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 966 bytes on disk
I20260808 01:50:01.955476 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":369,"lbm_reads_lt_1ms":4}
I20260808 01:50:01.977077 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 608 bytes on disk
I20260808 01:50:01.987002 5461 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45970 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=224, attempt_no=0}) took 1009 ms. Trace:
I20260808 01:50:01.991053 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":381,"lbm_reads_lt_1ms":4}
I20260808 01:50:02.005805 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.712s user 0.060s sys 0.000s Metrics: {"bytes_written":6818,"cfile_init":1,"dirs.queue_time_us":26964,"dirs.run_cpu_time_us":492,"dirs.run_wall_time_us":3329,"drs_written":1,"lbm_read_time_us":253,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":81,"thread_start_us":30190,"threads_started":1}
I20260808 01:50:02.015287 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.014976
I20260808 01:50:02.022336 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 906 bytes on disk
I20260808 01:50:02.024477 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4}
I20260808 01:50:02.087594 5414 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45970 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=226, attempt_no=0}) took 1107 ms. Trace:
I20260808 01:50:01.987538 5461 rpcz_store.cc:276] 0808 01:50:00.977670 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.977806 (+ 136us) service_pool.cc:224] Handling call
0808 01:50:01.986817 (+1009011us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.106329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9ee6f0c29217471e93394d4b9814144e
0808 01:50:01.122869 (+ 16540us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.122926 (+ 57us) write_op.cc:435] Acquired schema lock
0808 01:50:01.122939 (+ 13us) tablet.cc:662] Decoding operations
0808 01:50:01.123928 (+ 989us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.123991 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.124038 (+ 47us) tablet.cc:685] Acquiring locks for 4 operations
0808 01:50:01.124740 (+ 702us) tablet.cc:701] Row locks acquired
0808 01:50:01.124757 (+ 17us) write_op.cc:260] PREPARE: finished
0808 01:50:01.148753 (+ 23996us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b34000011f8. stack: 00007fc55a3f4cf3 00007fc55a3f4bf0 00007fc559fa2574 000000000050ff84 000000000050ff19 00007fc55f1c9e61 00007fc55f210bf4 00007fc561a11b95 00007fc561a11738 00007fc561a163d1 00007fc561a16389 00007fc561a16319 00007fc561a162e1 00007fc561a1550d 00000000005b6a64 00000000005b6899
0808 01:50:01.150286 (+ 1533us) write_op.cc:270] Start()
0808 01:50:01.157406 (+ 7120us) write_op.cc:276] Timestamp: P: 1786153801148835 usec, L: 0
0808 01:50:01.157458 (+ 52us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.158530 (+ 1072us) log.cc:844] Serialized 486 byte log entry
0808 01:50:01.539987 (+381457us) op_driver.cc:464] REPLICATION: finished
0808 01:50:01.813094 (+273107us) write_op.cc:301] APPLY: starting
0808 01:50:01.813162 (+ 68us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:01.813439 (+ 277us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:01.813490 (+ 51us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:01.814419 (+ 929us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:01.814606 (+ 187us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:01.814696 (+ 90us) write_op.cc:312] APPLY: finished
0808 01:50:01.815095 (+ 399us) log.cc:844] Serialized 40 byte log entry
0808 01:50:01.815550 (+ 455us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:01.815680 (+ 130us) write_op.cc:454] Released schema lock
0808 01:50:01.815776 (+ 96us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:01.986658 (+170882us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b34000011f8. stack: 00007fc55a3f4cf3 00007fc55a3f4bf0 00007fc559fa2574 000000000050ff84 000000000050ff19 00007fc55f1c9e61 00007fc55f211a3e 00007fc55f22558e 00007fc562609c61 00007fc561a13d8d 00007fc561a13533 00007fc561a17741 00007fc561a176f9 00007fc561a17689 00007fc561a17651 00007fc561a1687d
Metrics: {"child_traces":[["op",{"apply.queue_time_us":272891,"num_ops":4,"prepare.queue_time_us":69406,"prepare.run_cpu_time_us":6493,"prepare.run_wall_time_us":52721,"replication_time_us":382286,"spinlock_wait_cycles":130507776}]]}
I20260808 01:50:02.124336 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.978724
I20260808 01:50:02.088060 5414 rpcz_store.cc:276] 0808 01:50:00.979691 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.988562 (+ 8871us) service_pool.cc:224] Handling call
0808 01:50:02.078107 (+1089545us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.121668 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 213cfcc36a754065ab9203bf5a8f6f65
0808 01:50:01.122750 (+ 1082us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.122810 (+ 60us) write_op.cc:435] Acquired schema lock
0808 01:50:01.122822 (+ 12us) tablet.cc:662] Decoding operations
0808 01:50:01.124829 (+ 2007us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.124949 (+ 120us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.125008 (+ 59us) tablet.cc:685] Acquiring locks for 7 operations
0808 01:50:01.203375 (+ 78367us) tablet.cc:701] Row locks acquired
0808 01:50:01.203402 (+ 27us) write_op.cc:260] PREPARE: finished
0808 01:50:01.271929 (+ 68527us) write_op.cc:270] Start()
0808 01:50:01.272087 (+ 158us) write_op.cc:276] Timestamp: P: 1786153801271765 usec, L: 0
0808 01:50:01.272115 (+ 28us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.305565 (+ 33450us) log.cc:844] Serialized 616 byte log entry
0808 01:50:01.563121 (+257556us) op_driver.cc:464] REPLICATION: finished
0808 01:50:01.963245 (+400124us) write_op.cc:301] APPLY: starting
0808 01:50:01.963317 (+ 72us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:02.002926 (+ 39609us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:02.003004 (+ 78us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:02.030745 (+ 27741us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:02.030894 (+ 149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:02.030929 (+ 35us) write_op.cc:312] APPLY: finished
0808 01:50:02.031124 (+ 195us) log.cc:844] Serialized 58 byte log entry
0808 01:50:02.032911 (+ 1787us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.032995 (+ 84us) write_op.cc:454] Released schema lock
0808 01:50:02.033035 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:02.078012 (+ 44977us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b34000011f8. stack: 00007fc55a3f4cf3 00007fc55a3f4bf0 00007fc559fa2574 000000000050ff84 000000000050ff19 00007fc55f1c9e61 00007fc55f211a3e 00007fc55f22558e 00007fc562609c61 00007fc561a13d8d 00007fc561a13533 00007fc561a17741 00007fc561a176f9 00007fc561a17689 00007fc561a17651 00007fc561a1687d
Metrics: {"child_traces":[["op",{"apply.queue_time_us":399862,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"mutex_wait_us":29051,"num_ops":7,"prepare.queue_time_us":78633,"prepare.run_cpu_time_us":11351,"prepare.run_wall_time_us":184332,"replication_time_us":290785,"spinlock_wait_cycles":35596672}]]}
I20260808 01:50:02.243170 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 1.080s user 0.063s sys 0.011s Metrics: {"bytes_written":7130,"cfile_init":1,"dirs.queue_time_us":2793,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":12010,"drs_written":1,"lbm_read_time_us":328,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":95,"spinlock_wait_cycles":64120064}
W20260808 01:50:02.292878 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 140878us
I20260808 01:50:02.282529 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 1046 bytes on disk
I20260808 01:50:02.324679 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":342,"lbm_reads_lt_1ms":4}
W20260808 01:50:02.346642 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 93616.6us
I20260808 01:50:02.371771 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.014976
I20260808 01:50:02.286841 5414 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45970 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=230, attempt_no=0}) took 1118 ms. Trace:
I20260808 01:50:02.376222 5414 rpcz_store.cc:276] 0808 01:50:01.168868 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:01.177154 (+ 8286us) service_pool.cc:224] Handling call
0808 01:50:01.291190 (+114036us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b34000011f8. stack: 00007fc55a3f4cf3 00007fc55a3f4bf0 00007fc559fa2574 000000000050ff84 000000000050ff19 00007fc55f1c9e61 00007fc55f21021c 00007fc55f255e31 00007fc55f25912f 00007fc55f25a481 00007fc55f25a439 00007fc55f25a3c9 00007fc55f25a391 00007fc55f25968d 00000000005b6a64 00000000005b6899
0808 01:50:02.286768 (+995578us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.457894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 213cfcc36a754065ab9203bf5a8f6f65
0808 01:50:01.531851 (+ 73957us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.531923 (+ 72us) write_op.cc:435] Acquired schema lock
0808 01:50:01.531943 (+ 20us) tablet.cc:662] Decoding operations
0808 01:50:01.564677 (+ 32734us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.564758 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.564810 (+ 52us) tablet.cc:685] Acquiring locks for 6 operations
0808 01:50:01.565779 (+ 969us) tablet.cc:701] Row locks acquired
0808 01:50:01.565836 (+ 57us) write_op.cc:260] PREPARE: finished
0808 01:50:01.566120 (+ 284us) write_op.cc:270] Start()
0808 01:50:01.566331 (+ 211us) write_op.cc:276] Timestamp: P: 1786153801566040 usec, L: 0
0808 01:50:01.566385 (+ 54us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.567979 (+ 1594us) log.cc:844] Serialized 573 byte log entry
0808 01:50:02.017547 (+449568us) op_driver.cc:464] REPLICATION: finished
0808 01:50:02.177228 (+159681us) write_op.cc:301] APPLY: starting
0808 01:50:02.177284 (+ 56us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:02.284689 (+107405us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:02.284779 (+ 90us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:02.285731 (+ 952us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:02.285889 (+ 158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:02.285929 (+ 40us) write_op.cc:312] APPLY: finished
0808 01:50:02.286153 (+ 224us) log.cc:844] Serialized 52 byte log entry
0808 01:50:02.286362 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.286447 (+ 85us) write_op.cc:454] Released schema lock
0808 01:50:02.286487 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":21243776,"child_traces":[["op",{"apply.queue_time_us":159544,"mutex_wait_us":105840,"num_ops":6,"prepare.queue_time_us":129009,"prepare.run_cpu_time_us":5438,"prepare.run_wall_time_us":118187,"replication_time_us":451063}]]}
W20260808 01:50:02.432415 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 185797us
W20260808 01:50:02.573910 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.003s sys 0.000s
W20260808 01:50:02.892056 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.002s sys 0.000s
W20260808 01:50:03.085949 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 103276us
I20260808 01:50:03.104729 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.015s user 0.060s sys 0.003s Metrics: {"bytes_written":7340,"cfile_init":1,"dirs.queue_time_us":114343,"dirs.run_cpu_time_us":360,"dirs.run_wall_time_us":34631,"drs_written":1,"lbm_read_time_us":294,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":103,"thread_start_us":128799,"threads_started":1}
I20260808 01:50:03.168320 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.007081
W20260808 01:50:03.345841 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 74354.8us
I20260808 01:50:03.551971 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.179s user 0.057s sys 0.007s Metrics: {"bytes_written":7097,"cfile_init":1,"dirs.queue_time_us":85889,"dirs.run_cpu_time_us":438,"dirs.run_wall_time_us":26089,"drs_written":1,"lbm_read_time_us":629,"lbm_reads_lt_1ms":4,"lbm_write_time_us":30321,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"rows_written":93,"thread_start_us":9038,"threads_started":1}
I20260808 01:50:03.654335 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 1026 bytes on disk
I20260808 01:50:03.656355 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":309,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":60672}
W20260808 01:50:03.706383 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.108s user 0.001s sys 0.002s
I20260808 01:50:03.750564 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 192 bytes on disk
W20260808 01:50:03.751416 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 158075us
I20260808 01:50:03.790230 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.032s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":277,"lbm_reads_lt_1ms":4}
I20260808 01:50:03.806945 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.977371
W20260808 01:50:04.113638 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 63004.8us
I20260808 01:50:04.313082 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 2.185s user 0.080s sys 0.039s Metrics: {"bytes_written":7772,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2388,"cfile_cache_miss":26,"cfile_cache_miss_bytes":7306,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":177098,"dirs.run_cpu_time_us":3433,"dirs.run_wall_time_us":59093,"drs_written":1,"lbm_read_time_us":17638,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":42,"lbm_write_time_us":6966,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":119,"spinlock_wait_cycles":65920,"thread_start_us":295565,"threads_started":5}
I20260808 01:50:04.327112 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.030173
W20260808 01:50:04.507591 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.102s user 0.003s sys 0.000s
W20260808 01:50:04.681311 4873 thread.cc:641] dir /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.135s sys 0.141s
W20260808 01:50:04.718658 4873 thread.cc:608] dir /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data [worker] (thread pool) Time spent starting thread: real 0.809s user 0.135s sys 0.145s
W20260808 01:50:04.733914 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 206796us
W20260808 01:50:04.766525 4934 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.039s sys 0.061s
W20260808 01:50:04.766995 4934 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s user 0.039s sys 0.061s
I20260808 01:50:04.963310 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.107s user 0.081s sys 0.019s Metrics: {"bytes_written":8113,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2607,"cfile_cache_miss":26,"cfile_cache_miss_bytes":7885,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":42874,"dirs.run_cpu_time_us":2626,"dirs.run_wall_time_us":42486,"drs_written":1,"lbm_read_time_us":10633,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":3282,"lbm_writes_lt_1ms":23,"mutex_wait_us":45,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":131,"spinlock_wait_cycles":9879680,"thread_start_us":47705,"threads_started":6}
I20260808 01:50:04.970657 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.030173
W20260808 01:50:05.036803 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 140033us
I20260808 01:50:05.079207 5621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33802 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=345, attempt_no=0}) took 1022 ms. Trace:
W20260808 01:50:05.113857 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.001s sys 0.000s
I20260808 01:50:05.081103 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.911s user 0.263s sys 0.218s Metrics: {"bytes_written":8186,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2675,"cfile_cache_miss":26,"cfile_cache_miss_bytes":7959,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":376816,"dirs.run_cpu_time_us":3570,"dirs.run_wall_time_us":57497,"drs_written":1,"lbm_read_time_us":18058,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":47,"lbm_write_time_us":3233,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":135,"spinlock_wait_cycles":1436032,"thread_start_us":1364619,"threads_started":8}
I20260808 01:50:05.170735 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.030173
W20260808 01:50:05.300601 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 49877.5us
I20260808 01:50:05.354214 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.383s user 0.089s sys 0.001s Metrics: {"bytes_written":9547,"cfile_init":1,"dirs.queue_time_us":8042,"dirs.run_cpu_time_us":425,"dirs.run_wall_time_us":1629,"drs_written":1,"lbm_read_time_us":1669,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2686,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":185}
W20260808 01:50:05.361414 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 83439.2us
I20260808 01:50:05.364195 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 1934 bytes on disk
I20260808 01:50:05.133752 5621 rpcz_store.cc:276] 0808 01:50:04.055947 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:04.067407 (+ 11460us) service_pool.cc:224] Handling call
0808 01:50:05.078392 (+1010985us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:04.121314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2d7aa70246b445149411741e595f27c2
0808 01:50:04.122297 (+ 983us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:04.122329 (+ 32us) write_op.cc:435] Acquired schema lock
0808 01:50:04.122357 (+ 28us) tablet.cc:662] Decoding operations
0808 01:50:04.123195 (+ 838us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:04.123281 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:04.123377 (+ 96us) tablet.cc:685] Acquiring locks for 5 operations
0808 01:50:04.124293 (+ 916us) tablet.cc:701] Row locks acquired
0808 01:50:04.124334 (+ 41us) write_op.cc:260] PREPARE: finished
0808 01:50:04.124560 (+ 226us) write_op.cc:270] Start()
0808 01:50:04.124699 (+ 139us) write_op.cc:276] Timestamp: P: 1786153804124451 usec, L: 0
0808 01:50:04.124764 (+ 65us) op_driver.cc:348] REPLICATION: starting
0808 01:50:04.489610 (+364846us) log.cc:844] Serialized 530 byte log entry
0808 01:50:04.928755 (+439145us) op_driver.cc:464] REPLICATION: finished
0808 01:50:04.966105 (+ 37350us) write_op.cc:301] APPLY: starting
0808 01:50:04.966163 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:04.966800 (+ 637us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:04.968722 (+ 1922us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:05.063673 (+ 94951us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:05.063924 (+ 251us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:05.063973 (+ 49us) write_op.cc:312] APPLY: finished
0808 01:50:05.064214 (+ 241us) log.cc:844] Serialized 46 byte log entry
0808 01:50:05.077553 (+ 13339us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:05.077716 (+ 163us) write_op.cc:454] Released schema lock
0808 01:50:05.077777 (+ 61us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37194,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"num_ops":5,"prepare.queue_time_us":48799,"prepare.run_cpu_time_us":5058,"prepare.run_wall_time_us":370879,"replication_time_us":803843}]]}
I20260808 01:50:05.393121 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.028s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2930,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20260808 01:50:05.401378 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=0.949413
W20260808 01:50:05.490633 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45737.1us
W20260808 01:50:05.886130 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 110642us
I20260808 01:50:05.913408 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.563s user 0.082s sys 0.005s Metrics: {"bytes_written":8794,"cfile_init":1,"dirs.queue_time_us":28776,"dirs.run_cpu_time_us":423,"dirs.run_wall_time_us":13347,"drs_written":1,"lbm_read_time_us":331,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":158,"thread_start_us":29042,"threads_started":1}
I20260808 01:50:06.057987 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 1866 bytes on disk
I20260808 01:50:06.060452 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":571,"lbm_reads_lt_1ms":8}
I20260808 01:50:06.086584 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.673s user 0.104s sys 0.002s Metrics: {"bytes_written":9665,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3725,"cfile_cache_miss":26,"cfile_cache_miss_bytes":11165,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":70760,"dirs.run_cpu_time_us":2588,"dirs.run_wall_time_us":30867,"drs_written":1,"lbm_read_time_us":27503,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":30629,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":192,"thread_start_us":84202,"threads_started":4}
W20260808 01:50:06.093065 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 47417.5us
I20260808 01:50:06.100347 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 1994 bytes on disk
I20260808 01:50:06.130832 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4}
I20260808 01:50:06.134923 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.953s user 0.076s sys 0.011s Metrics: {"bytes_written":9222,"cfile_init":1,"dirs.queue_time_us":13872,"dirs.run_cpu_time_us":326,"dirs.run_wall_time_us":7419,"drs_written":1,"lbm_read_time_us":267,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":174,"thread_start_us":25220,"threads_started":1}
I20260808 01:50:06.136399 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.030173
I20260808 01:50:06.168452 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.030173
I20260808 01:50:06.304212 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 1281 bytes on disk
I20260808 01:50:06.382755 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4}
W20260808 01:50:06.387583 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 60504.7us
I20260808 01:50:06.514456 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=0.950432
I20260808 01:50:06.514633 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.345s user 0.076s sys 0.001s Metrics: {"bytes_written":8811,"cfile_init":1,"dirs.queue_time_us":4009,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1449,"drs_written":1,"lbm_read_time_us":274,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8589,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":159}
I20260808 01:50:06.538544 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 1844 bytes on disk
W20260808 01:50:06.539445 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 103187us
I20260808 01:50:06.540481 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4}
I20260808 01:50:06.583730 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.012778
W20260808 01:50:06.600824 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 112385us
W20260808 01:50:06.754973 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 95238.9us
I20260808 01:50:06.818917 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.508s user 0.070s sys 0.003s Metrics: {"bytes_written":8548,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1483,"drs_written":1,"lbm_read_time_us":261,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":148}
I20260808 01:50:06.883921 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 1570 bytes on disk
I20260808 01:50:06.886277 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":273,"lbm_reads_lt_1ms":4}
I20260808 01:50:06.900815 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 1404 bytes on disk
I20260808 01:50:06.903195 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":405,"lbm_reads_lt_1ms":4}
I20260808 01:50:06.911842 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.009656
W20260808 01:50:06.969936 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 100477us
W20260808 01:50:06.995558 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 69742.4us
W20260808 01:50:07.007004 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 50291.3us
W20260808 01:50:07.148921 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 64187.7us
W20260808 01:50:07.289255 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 85652.1us
W20260808 01:50:07.360674 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45021.8us
I20260808 01:50:07.443583 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.852s user 0.098s sys 0.001s Metrics: {"bytes_written":10092,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4011,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12087,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25436,"dirs.run_cpu_time_us":639,"dirs.run_wall_time_us":36413,"drs_written":1,"lbm_read_time_us":144169,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":48,"lbm_write_time_us":2642,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":208,"spinlock_wait_cycles":24320,"thread_start_us":24330,"threads_started":1}
I20260808 01:50:07.468079 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 2167 bytes on disk
I20260808 01:50:07.470362 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":336,"lbm_reads_lt_1ms":4}
I20260808 01:50:07.477003 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 1436 bytes on disk
I20260808 01:50:07.478699 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4}
I20260808 01:50:07.507294 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 548 bytes on disk
I20260808 01:50:07.507813 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.595s user 0.098s sys 0.005s Metrics: {"bytes_written":10593,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4320,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12986,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":16924,"dirs.run_cpu_time_us":369,"dirs.run_wall_time_us":11213,"drs_written":1,"lbm_read_time_us":3713,"lbm_reads_lt_1ms":46,"lbm_write_time_us":12519,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":225,"thread_start_us":17158,"threads_started":1}
I20260808 01:50:07.519716 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.030173
W20260808 01:50:07.550623 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.002s sys 0.000s
I20260808 01:50:07.623090 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4}
W20260808 01:50:07.636412 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.004s sys 0.000s
I20260808 01:50:07.719619 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.030173
W20260808 01:50:07.910077 5389 log.cc:927] Time spent T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260808 01:50:08.125413 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.610s user 0.138s sys 0.063s Metrics: {"bytes_written":8994,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3231,"cfile_cache_miss":26,"cfile_cache_miss_bytes":9633,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":286670,"dirs.run_cpu_time_us":2837,"dirs.run_wall_time_us":97669,"drs_written":1,"lbm_read_time_us":93167,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":44,"lbm_write_time_us":19085,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":165,"thread_start_us":288232,"threads_started":6}
W20260808 01:50:08.138880 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 58668.4us
I20260808 01:50:08.138962 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 1739 bytes on disk
W20260808 01:50:08.139976 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 52676.7us
I20260808 01:50:08.141708 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":348,"lbm_reads_lt_1ms":4}
I20260808 01:50:08.147140 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.030173
W20260808 01:50:08.154727 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 288645us
I20260808 01:50:08.210801 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.684s user 0.086s sys 0.004s Metrics: {"bytes_written":9220,"cfile_init":1,"dirs.queue_time_us":14664,"dirs.run_cpu_time_us":307,"dirs.run_wall_time_us":912,"drs_written":1,"lbm_read_time_us":6814,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":5734,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":174}
I20260808 01:50:08.223431 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 2331 bytes on disk
I20260808 01:50:08.228199 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":390,"lbm_reads_lt_1ms":4}
I20260808 01:50:08.247493 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.983953
I20260808 01:50:08.603459 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.854s user 0.089s sys 0.009s Metrics: {"bytes_written":9220,"cfile_init":1,"dirs.queue_time_us":14926,"dirs.run_cpu_time_us":457,"dirs.run_wall_time_us":15620,"drs_written":1,"lbm_read_time_us":338,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":174,"spinlock_wait_cycles":3093248,"thread_start_us":13583,"threads_started":1}
W20260808 01:50:08.699570 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 92889.9us
I20260808 01:50:08.621400 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.983953
I20260808 01:50:08.830328 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.682s user 0.088s sys 0.011s Metrics: {"bytes_written":10725,"cfile_init":1,"dirs.queue_time_us":95,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":917,"drs_written":1,"lbm_read_time_us":283,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3622,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":232}
I20260808 01:50:08.839731 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 2391 bytes on disk
W20260808 01:50:08.845708 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 50326.9us
I20260808 01:50:08.890995 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":353,"lbm_reads_lt_1ms":4}
I20260808 01:50:08.916033 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.004889
W20260808 01:50:08.994994 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 86213.1us
W20260808 01:50:09.050570 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 70569.2us
I20260808 01:50:09.109831 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.845s user 0.106s sys 0.022s Metrics: {"bytes_written":10408,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4213,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12637,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":48896,"dirs.run_cpu_time_us":2410,"dirs.run_wall_time_us":68348,"drs_written":1,"lbm_read_time_us":38710,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":48,"lbm_write_time_us":3058,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":219,"thread_start_us":91150,"threads_started":6}
I20260808 01:50:09.142297 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.060569
W20260808 01:50:09.533718 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 49227.3us
I20260808 01:50:09.561551 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.418s user 0.111s sys 0.009s Metrics: {"bytes_written":12291,"cfile_init":1,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":389,"dirs.run_wall_time_us":1486,"drs_written":1,"lbm_read_time_us":315,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2808,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":290}
I20260808 01:50:09.567423 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 2949 bytes on disk
I20260808 01:50:09.569233 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4}
I20260808 01:50:09.574759 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.030173
I20260808 01:50:10.054623 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.479s user 0.069s sys 0.006s Metrics: {"bytes_written":8570,"cfile_init":1,"dirs.queue_time_us":56976,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":7417,"drs_written":1,"lbm_read_time_us":255,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3446,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":149,"thread_start_us":78626,"threads_started":1}
I20260808 01:50:10.062923 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.013020
W20260808 01:50:10.055033 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 85041.5us
W20260808 01:50:10.257771 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 45709us
I20260808 01:50:10.806344 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 2.062s user 0.107s sys 0.013s Metrics: {"bytes_written":10408,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4213,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12637,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":218490,"dirs.run_cpu_time_us":3060,"dirs.run_wall_time_us":114294,"drs_written":1,"lbm_read_time_us":3825,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2755,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":219,"thread_start_us":427691,"threads_started":8}
I20260808 01:50:10.814188 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 2283 bytes on disk
I20260808 01:50:10.817276 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.753s user 0.146s sys 0.059s Metrics: {"bytes_written":11975,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5325,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15985,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":230103,"dirs.run_cpu_time_us":3622,"dirs.run_wall_time_us":80606,"drs_written":1,"lbm_read_time_us":16316,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":4114,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":5,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":280,"spinlock_wait_cycles":30848,"thread_start_us":314037,"threads_started":5}
I20260808 01:50:10.827867 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.060569
I20260808 01:50:10.815917 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":285,"lbm_reads_lt_1ms":4}
I20260808 01:50:10.859074 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.060569
W20260808 01:50:11.080230 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 70256.6us
I20260808 01:50:11.187801 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 2.266s user 0.117s sys 0.002s Metrics: {"bytes_written":11554,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5028,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14908,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":128534,"dirs.run_cpu_time_us":2436,"dirs.run_wall_time_us":81906,"drs_written":1,"lbm_read_time_us":187833,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":42947,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":264,"thread_start_us":412066,"threads_started":6}
I20260808 01:50:11.194597 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 2701 bytes on disk
I20260808 01:50:11.196444 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4}
W20260808 01:50:11.199030 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.133s user 0.004s sys 0.000s
I20260808 01:50:11.202577 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.060569
W20260808 01:50:11.285884 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 107232us
I20260808 01:50:11.358074 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.155s user 0.118s sys 0.000s Metrics: {"bytes_written":13173,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":348,"dirs.run_wall_time_us":19345,"drs_written":1,"lbm_read_time_us":260,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":325}
I20260808 01:50:11.364107 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 3299 bytes on disk
I20260808 01:50:11.365911 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4}
I20260808 01:50:11.372452 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.060569
W20260808 01:50:11.553728 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 83241.3us
I20260808 01:50:11.658016 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.285s user 0.135s sys 0.002s Metrics: {"bytes_written":14356,"cfile_init":1,"dirs.queue_time_us":680,"dirs.run_cpu_time_us":369,"dirs.run_wall_time_us":2535,"drs_written":1,"lbm_read_time_us":323,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":370}
I20260808 01:50:11.667771 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 3690 bytes on disk
I20260808 01:50:11.669498 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":286,"lbm_reads_lt_1ms":4}
W20260808 01:50:11.672188 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 104113us
I20260808 01:50:11.675419 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.060569
W20260808 01:50:11.726539 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 45012.5us
W20260808 01:50:11.748561 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 70517.5us
W20260808 01:50:11.933728 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 61194.4us
I20260808 01:50:12.022636 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.346s user 0.126s sys 0.004s Metrics: {"bytes_written":13547,"cfile_init":1,"dirs.queue_time_us":6867,"dirs.run_cpu_time_us":370,"dirs.run_wall_time_us":10953,"drs_written":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":338,"thread_start_us":7114,"threads_started":1}
I20260808 01:50:12.030303 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.060569
W20260808 01:50:12.082906 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 44245.4us
W20260808 01:50:12.174484 5270 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 76437.5us
W20260808 01:50:12.325711 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 118401us
I20260808 01:50:12.491860 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.461s user 0.111s sys 0.004s Metrics: {"bytes_written":12277,"cfile_init":1,"dirs.queue_time_us":6693,"dirs.run_cpu_time_us":342,"dirs.run_wall_time_us":2904,"drs_written":1,"lbm_read_time_us":372,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":289}
I20260808 01:50:12.498446 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 3427 bytes on disk
I20260808 01:50:12.555019 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4}
W20260808 01:50:12.557008 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 67502.2us
I20260808 01:50:12.583499 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 2983 bytes on disk
I20260808 01:50:12.606536 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":379,"lbm_reads_lt_1ms":4}
I20260808 01:50:12.676319 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.980824
W20260808 01:50:12.793929 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 209076us
I20260808 01:50:12.811956 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 1.983s user 0.121s sys 0.003s Metrics: {"bytes_written":12290,"cfile_init":1,"dirs.queue_time_us":35911,"dirs.run_cpu_time_us":555,"dirs.run_wall_time_us":15411,"drs_written":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2950,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":290,"thread_start_us":37994,"threads_started":1}
I20260808 01:50:12.820115 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.060569
I20260808 01:50:12.838750 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 1.979s user 0.122s sys 0.009s Metrics: {"bytes_written":12732,"cfile_init":1,"dirs.queue_time_us":240219,"dirs.run_cpu_time_us":428,"dirs.run_wall_time_us":16224,"drs_written":1,"lbm_read_time_us":267,"lbm_reads_lt_1ms":4,"lbm_write_time_us":23569,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":307,"thread_start_us":150731,"threads_started":1}
I20260808 01:50:12.865509 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=0.546045
W20260808 01:50:13.116067 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 154851us
W20260808 01:50:13.250164 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 54334.5us
I20260808 01:50:13.301725 5621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33802 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=720, attempt_no=0}) took 1003 ms. Trace:
I20260808 01:50:13.315198 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.489s user 0.133s sys 0.019s Metrics: {"bytes_written":12786,"cfile_init":1,"dirs.queue_time_us":52557,"dirs.run_cpu_time_us":408,"dirs.run_wall_time_us":11989,"drs_written":1,"lbm_read_time_us":346,"lbm_reads_lt_1ms":4,"lbm_write_time_us":29808,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":311,"thread_start_us":52820,"threads_started":1}
I20260808 01:50:13.339865 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 3158 bytes on disk
I20260808 01:50:13.302258 5621 rpcz_store.cc:276] 0808 01:50:12.224634 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.225899 (+ 1265us) service_pool.cc:224] Handling call
0808 01:50:13.227833 (+1001934us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:12.235668 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2d7aa70246b445149411741e595f27c2
0808 01:50:12.236553 (+ 885us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:12.236621 (+ 68us) write_op.cc:435] Acquired schema lock
0808 01:50:12.236678 (+ 57us) tablet.cc:662] Decoding operations
0808 01:50:12.237430 (+ 752us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:12.237480 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:12.237527 (+ 47us) tablet.cc:685] Acquiring locks for 5 operations
0808 01:50:12.238435 (+ 908us) tablet.cc:701] Row locks acquired
0808 01:50:12.238458 (+ 23us) write_op.cc:260] PREPARE: finished
0808 01:50:12.238814 (+ 356us) write_op.cc:270] Start()
0808 01:50:12.238954 (+ 140us) write_op.cc:276] Timestamp: P: 1786153812238744 usec, L: 0
0808 01:50:12.239018 (+ 64us) op_driver.cc:348] REPLICATION: starting
0808 01:50:12.240085 (+ 1067us) log.cc:844] Serialized 530 byte log entry
0808 01:50:13.084839 (+844754us) op_driver.cc:464] REPLICATION: finished
0808 01:50:13.151660 (+ 66821us) write_op.cc:301] APPLY: starting
0808 01:50:13.151722 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:13.155288 (+ 3566us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:13.155417 (+ 129us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:13.156456 (+ 1039us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:13.156678 (+ 222us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:13.156751 (+ 73us) write_op.cc:312] APPLY: finished
0808 01:50:13.157114 (+ 363us) log.cc:844] Serialized 46 byte log entry
0808 01:50:13.227465 (+ 70351us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:13.227541 (+ 76us) write_op.cc:454] Released schema lock
0808 01:50:13.227576 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56501,"cfile_cache_hit":4,"cfile_cache_hit_bytes":8254,"num_ops":5,"prepare.queue_time_us":8260,"prepare.run_cpu_time_us":4941,"prepare.run_wall_time_us":21021,"replication_time_us":830158,"thread_start_us":10905,"threads_started":1}]]}
I20260808 01:50:13.342519 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4}
I20260808 01:50:13.395624 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 2975 bytes on disk
I20260808 01:50:13.406675 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":288,"lbm_reads_lt_1ms":4}
W20260808 01:50:13.427074 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 174657us
I20260808 01:50:13.504496 5792 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33802 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=728, attempt_no=0}) took 1060 ms. Trace:
I20260808 01:50:13.515028 5814 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33802 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=718, attempt_no=0}) took 1293 ms. Trace:
I20260808 01:50:13.531240 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.836s user 0.128s sys 0.011s Metrics: {"bytes_written":13380,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6314,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18926,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":92249,"dirs.run_cpu_time_us":2420,"dirs.run_wall_time_us":13165,"drs_written":1,"lbm_read_time_us":9914,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":3061,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":334,"spinlock_wait_cycles":37248,"thread_start_us":105100,"threads_started":5}
I20260808 01:50:13.538940 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 3397 bytes on disk
I20260808 01:50:13.543818 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":350,"lbm_reads_lt_1ms":4}
I20260808 01:50:13.546546 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.013176
I20260808 01:50:13.552078 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.060569
I20260808 01:50:13.575067 5621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33802 (ReqId={client: 33bc24b9a653469bb09255c6c80ddbe0, seq_no=740, attempt_no=0}) took 1054 ms. Trace:
I20260808 01:50:13.575578 5621 rpcz_store.cc:276] 0808 01:50:12.520777 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.520935 (+ 158us) service_pool.cc:224] Handling call
0808 01:50:12.586591 (+ 65656us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b34000011f8. stack: 00007ff2ff796cf3 00007ff2ff796bf0 00007ff2ff344574 000000000050ff84 000000000050ff19 00007ff30456be61 00007ff3045b221c 00007ff3045f7e31 00007ff3045fb12f 00007ff3045fc481 00007ff3045fc439 00007ff3045fc3c9 00007ff3045fc391 00007ff3045fb68d 00000000005b6a64 00000000005b6899
0808 01:50:13.574981 (+988390us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:12.587762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8d367581a54c433ea4f227bd4f479ed2
0808 01:50:12.588227 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:12.588253 (+ 26us) write_op.cc:435] Acquired schema lock
0808 01:50:12.588269 (+ 16us) tablet.cc:662] Decoding operations
0808 01:50:12.588723 (+ 454us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:12.588759 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:12.588785 (+ 26us) tablet.cc:685] Acquiring locks for 6 operations
0808 01:50:12.589217 (+ 432us) tablet.cc:701] Row locks acquired
0808 01:50:12.589234 (+ 17us) write_op.cc:260] PREPARE: finished
0808 01:50:12.589394 (+ 160us) write_op.cc:270] Start()
0808 01:50:12.589481 (+ 87us) write_op.cc:276] Timestamp: P: 1786153812589314 usec, L: 0
0808 01:50:12.589501 (+ 20us) op_driver.cc:348] REPLICATION: starting
0808 01:50:12.590145 (+ 644us) log.cc:844] Serialized 573 byte log entry
0808 01:50:13.298484 (+708339us) op_driver.cc:464] REPLICATION: finished
0808 01:50:13.473230 (+174746us) write_op.cc:301] APPLY: starting
0808 01:50:13.473311 (+ 81us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:13.473499 (+ 188us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:13.473600 (+ 101us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:13.474556 (+ 956us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:13.474761 (+ 205us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:13.474817 (+ 56us) write_op.cc:312] APPLY: finished
0808 01:50:13.475153 (+ 336us) log.cc:844] Serialized 52 byte log entry
0808 01:50:13.574449 (+ 99296us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:13.574567 (+ 118us) write_op.cc:454] Released schema lock
0808 01:50:13.574619 (+ 52us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":144831360,"child_traces":[["op",{"apply.queue_time_us":174590,"num_ops":6,"prepare.queue_time_us":169,"prepare.run_cpu_time_us":4594,"prepare.run_wall_time_us":53893,"raft.queue_time_us":38041,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":11,"replication_time_us":708908,"thread_start_us":51070,"threads_started":1}]]}
I20260808 01:50:13.515542 5814 rpcz_store.cc:276] 0808 01:50:12.221747 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.227716 (+ 5969us) service_pool.cc:224] Handling call
0808 01:50:13.514958 (+1287242us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:12.243846 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8d367581a54c433ea4f227bd4f479ed2
0808 01:50:12.273230 (+ 29384us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:12.273259 (+ 29us) write_op.cc:435] Acquired schema lock
0808 01:50:12.273272 (+ 13us) tablet.cc:662] Decoding operations
0808 01:50:12.327697 (+ 54425us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:12.327755 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:12.327778 (+ 23us) tablet.cc:685] Acquiring locks for 5 operations
0808 01:50:12.328531 (+ 753us) tablet.cc:701] Row locks acquired
0808 01:50:12.328547 (+ 16us) write_op.cc:260] PREPARE: finished
0808 01:50:12.330303 (+ 1756us) write_op.cc:270] Start()
0808 01:50:12.330399 (+ 96us) write_op.cc:276] Timestamp: P: 1786153812330243 usec, L: 0
0808 01:50:12.330415 (+ 16us) op_driver.cc:348] REPLICATION: starting
0808 01:50:12.356933 (+ 26518us) log.cc:844] Serialized 530 byte log entry
0808 01:50:13.296850 (+939917us) op_driver.cc:464] REPLICATION: finished
0808 01:50:13.485772 (+188922us) write_op.cc:301] APPLY: starting
0808 01:50:13.485886 (+ 114us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:13.486109 (+ 223us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:13.486244 (+ 135us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:13.487298 (+ 1054us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:13.513087 (+ 25789us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:13.513165 (+ 78us) write_op.cc:312] APPLY: finished
0808 01:50:13.513526 (+ 361us) log.cc:844] Serialized 46 byte log entry
0808 01:50:13.514519 (+ 993us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:13.514609 (+ 90us) write_op.cc:454] Released schema lock
0808 01:50:13.514651 (+ 42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149044,"num_ops":5,"prepare.queue_time_us":14295,"prepare.run_cpu_time_us":2532,"prepare.run_wall_time_us":113295,"replication_time_us":963016}]]}
I20260808 01:50:13.505023 5792 rpcz_store.cc:276] 0808 01:50:12.432918 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.433553 (+ 635us) service_pool.cc:224] Handling call
0808 01:50:13.492734 (+1059181us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:12.434824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8d367581a54c433ea4f227bd4f479ed2
0808 01:50:12.435281 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:12.435309 (+ 28us) write_op.cc:435] Acquired schema lock
0808 01:50:12.435319 (+ 10us) tablet.cc:662] Decoding operations
0808 01:50:12.435619 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:12.435661 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:12.435691 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:50:12.435899 (+ 208us) tablet.cc:701] Row locks acquired
0808 01:50:12.435917 (+ 18us) write_op.cc:260] PREPARE: finished
0808 01:50:12.436080 (+ 163us) write_op.cc:270] Start()
0808 01:50:12.436174 (+ 94us) write_op.cc:276] Timestamp: P: 1786153812436019 usec, L: 0
0808 01:50:12.436195 (+ 21us) op_driver.cc:348] REPLICATION: starting
0808 01:50:12.436779 (+ 584us) log.cc:844] Serialized 399 byte log entry
0808 01:50:13.298289 (+861510us) op_driver.cc:464] REPLICATION: finished
0808 01:50:13.467592 (+169303us) write_op.cc:301] APPLY: starting
0808 01:50:13.467677 (+ 85us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:13.467824 (+ 147us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:13.467953 (+ 129us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:13.468477 (+ 524us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:13.468666 (+ 189us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0808 01:50:13.468730 (+ 64us) write_op.cc:312] APPLY: finished
0808 01:50:13.468998 (+ 268us) log.cc:844] Serialized 28 byte log entry
0808 01:50:13.492282 (+ 23284us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:13.492352 (+ 70us) write_op.cc:454] Released schema lock
0808 01:50:13.492394 (+ 42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":169137,"num_ops":2,"prepare.queue_time_us":141,"prepare.run_cpu_time_us":2222,"prepare.run_wall_time_us":3449,"replication_time_us":862004}]]}
W20260808 01:50:13.848523 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 57736.2us
W20260808 01:50:14.306195 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45757.7us
W20260808 01:50:14.351130 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 44151.4us
W20260808 01:50:14.462431 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 109721us
I20260808 01:50:14.606364 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 1.059s user 0.136s sys 0.028s Metrics: {"bytes_written":14055,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6786,"cfile_cache_miss":26,"cfile_cache_miss_bytes":20222,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":266740,"dirs.run_cpu_time_us":2858,"dirs.run_wall_time_us":73788,"drs_written":1,"lbm_read_time_us":3362,"lbm_reads_lt_1ms":46,"lbm_write_time_us":16725,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":360,"spinlock_wait_cycles":167065984,"thread_start_us":566296,"threads_started":6}
I20260808 01:50:14.614166 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.060569
I20260808 01:50:14.620036 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 1.723s user 0.136s sys 0.043s Metrics: {"bytes_written":12790,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5881,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17581,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":451428,"dirs.run_cpu_time_us":3053,"dirs.run_wall_time_us":15102,"drs_written":1,"lbm_read_time_us":4147,"lbm_reads_lt_1ms":50,"lbm_write_time_us":17278,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":310,"thread_start_us":489715,"threads_started":5}
I20260808 01:50:14.627314 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.060569
I20260808 01:50:14.628201 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 1.075s user 0.125s sys 0.022s Metrics: {"bytes_written":13119,"cfile_init":1,"dirs.queue_time_us":116218,"dirs.run_cpu_time_us":332,"dirs.run_wall_time_us":10352,"drs_written":1,"lbm_read_time_us":324,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2732,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":322,"thread_start_us":100879,"threads_started":1}
W20260808 01:50:14.630601 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 159509us
I20260808 01:50:14.655957 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.013111
W20260808 01:50:14.857981 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 53557.9us
W20260808 01:50:15.469369 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 115996us
W20260808 01:50:15.698879 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.004s sys 0.000s
I20260808 01:50:15.776149 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 1.161s user 0.149s sys 0.030s Metrics: {"bytes_written":13415,"cfile_init":1,"dirs.queue_time_us":180674,"dirs.run_cpu_time_us":350,"dirs.run_wall_time_us":6293,"drs_written":1,"lbm_read_time_us":300,"lbm_reads_lt_1ms":4,"lbm_write_time_us":16151,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":335,"thread_start_us":179997,"threads_started":1}
I20260808 01:50:15.852126 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.977997
I20260808 01:50:15.971171 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.343s user 0.130s sys 0.014s Metrics: {"bytes_written":13151,"cfile_init":1,"dirs.queue_time_us":118822,"dirs.run_cpu_time_us":352,"dirs.run_wall_time_us":18005,"drs_written":1,"lbm_read_time_us":272,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2658,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":324,"thread_start_us":118921,"threads_started":1}
W20260808 01:50:16.025738 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 79797.6us
I20260808 01:50:16.073763 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.010800
W20260808 01:50:16.209950 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.155s user 0.003s sys 0.000s
W20260808 01:50:16.445799 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 56035.3us
W20260808 01:50:16.519951 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 71599.9us
W20260808 01:50:16.561219 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 45143.3us
W20260808 01:50:16.929706 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.003s sys 0.000s
I20260808 01:50:17.065996 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 2.409s user 0.131s sys 0.000s Metrics: {"bytes_written":14673,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7207,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21449,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":221229,"dirs.run_cpu_time_us":2746,"dirs.run_wall_time_us":53845,"drs_written":1,"lbm_read_time_us":101130,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":48,"lbm_write_time_us":8176,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":831,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":383,"spinlock_wait_cycles":46336,"thread_start_us":357378,"threads_started":5}
I20260808 01:50:17.077226 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 3817 bytes on disk
I20260808 01:50:17.084319 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":297,"lbm_reads_lt_1ms":4}
I20260808 01:50:17.095782 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.060569
W20260808 01:50:17.407553 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.003s
W20260808 01:50:17.411293 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 43911.7us
W20260808 01:50:17.594529 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 77582.8us
W20260808 01:50:17.688977 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 68218us
W20260808 01:50:17.833181 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 83800.2us
W20260808 01:50:17.970299 5011 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.249s user 0.004s sys 0.000s
I20260808 01:50:18.292783 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 2.214s user 0.369s sys 0.347s Metrics: {"bytes_written":15404,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7730,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22904,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":573269,"dirs.run_cpu_time_us":2886,"dirs.run_wall_time_us":92728,"drs_written":1,"lbm_read_time_us":78643,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":48,"lbm_write_time_us":3182,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":412,"thread_start_us":1030058,"threads_started":5}
I20260808 01:50:18.301571 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
W20260808 01:50:18.395525 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 103768us
W20260808 01:50:18.471118 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 77837.3us
I20260808 01:50:18.543659 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 2.686s user 0.164s sys 0.030s Metrics: {"bytes_written":14993,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7404,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21986,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":257402,"dirs.run_cpu_time_us":2583,"dirs.run_wall_time_us":90527,"drs_written":1,"lbm_read_time_us":233300,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":48,"lbm_write_time_us":174877,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":20,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":394,"thread_start_us":279483,"threads_started":7}
I20260808 01:50:18.551644 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
I20260808 01:50:18.469128 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.373s user 0.124s sys 0.007s Metrics: {"bytes_written":14374,"cfile_init":1,"dirs.queue_time_us":333215,"dirs.run_cpu_time_us":416,"dirs.run_wall_time_us":19047,"drs_written":1,"lbm_read_time_us":329,"lbm_reads_lt_1ms":4,"lbm_write_time_us":99407,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":370,"thread_start_us":120715,"threads_started":1}
I20260808 01:50:18.609861 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.012862
W20260808 01:50:18.798472 5147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 75445.4us
W20260808 01:50:18.862488 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.215s user 0.002s sys 0.000s
I20260808 01:50:18.857151 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.304s user 0.104s sys 0.009s Metrics: {"bytes_written":12537,"cfile_init":1,"dirs.queue_time_us":6927,"dirs.run_cpu_time_us":363,"dirs.run_wall_time_us":2932,"drs_written":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":300}
I20260808 01:50:18.876228 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 3946 bytes on disk
I20260808 01:50:18.878218 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":379,"lbm_reads_lt_1ms":4}
I20260808 01:50:18.885773 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 3607 bytes on disk
I20260808 01:50:18.891867 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":319,"lbm_reads_lt_1ms":4}
I20260808 01:50:18.901885 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.013925
W20260808 01:50:19.034111 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 111533us
W20260808 01:50:19.212970 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 145752us
W20260808 01:50:19.739010 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 49413.4us
I20260808 01:50:19.779433 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.873s user 0.231s sys 0.103s Metrics: {"bytes_written":18361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9796,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29196,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":301813,"dirs.run_cpu_time_us":2670,"dirs.run_wall_time_us":33307,"drs_written":1,"lbm_read_time_us":3918,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2991,"lbm_writes_lt_1ms":23,"mutex_wait_us":313,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":525,"spinlock_wait_cycles":31621504,"thread_start_us":387495,"threads_started":5}
I20260808 01:50:19.792377 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 5155 bytes on disk
I20260808 01:50:19.799690 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":318,"lbm_reads_lt_1ms":4}
I20260808 01:50:19.815289 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.060569
I20260808 01:50:19.943558 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.320s user 0.146s sys 0.006s Metrics: {"bytes_written":17475,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9139,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27113,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":68554,"dirs.run_cpu_time_us":384,"dirs.run_wall_time_us":22104,"drs_written":1,"lbm_read_time_us":21306,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":11383,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":489,"thread_start_us":86561,"threads_started":1}
I20260808 01:50:19.975374 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 4809 bytes on disk
I20260808 01:50:19.977190 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":281,"lbm_reads_lt_1ms":4}
I20260808 01:50:19.982472 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
I20260808 01:50:20.102778 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.277s user 0.106s sys 0.000s Metrics: {"bytes_written":12368,"cfile_init":1,"dirs.queue_time_us":102,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":986,"drs_written":1,"lbm_read_time_us":13592,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":4236,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":294}
I20260808 01:50:20.114606 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 5306 bytes on disk
I20260808 01:50:20.117340 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":582,"lbm_reads_lt_1ms":8}
I20260808 01:50:20.131597 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 2868 bytes on disk
I20260808 01:50:20.133437 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4}
I20260808 01:50:20.142625 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=1.011745
W20260808 01:50:20.201717 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 51448.7us
W20260808 01:50:20.402019 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.002s sys 0.000s
W20260808 01:50:20.462580 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 53174.5us
I20260808 01:50:20.514269 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 2.209s user 0.153s sys 0.024s Metrics: {"bytes_written":14850,"cfile_init":1,"dirs.queue_time_us":284475,"dirs.run_cpu_time_us":388,"dirs.run_wall_time_us":33065,"drs_written":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4,"lbm_write_time_us":162235,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"rows_written":390,"thread_start_us":212261,"threads_started":1}
I20260808 01:50:20.536118 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 4094 bytes on disk
I20260808 01:50:20.542924 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":304,"lbm_reads_lt_1ms":4}
I20260808 01:50:20.548800 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.013954
I20260808 01:50:20.607640 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.464s user 0.156s sys 0.007s Metrics: {"bytes_written":18077,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28543,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":50728,"dirs.run_cpu_time_us":336,"dirs.run_wall_time_us":9843,"drs_written":1,"lbm_read_time_us":3539,"lbm_reads_lt_1ms":46,"lbm_write_time_us":3045,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":513,"thread_start_us":51221,"threads_started":1}
I20260808 01:50:20.618973 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.060569
W20260808 01:50:20.734910 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 45454.4us
I20260808 01:50:20.848336 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.860s user 0.114s sys 0.006s Metrics: {"bytes_written":13124,"cfile_init":1,"dirs.queue_time_us":57296,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":1867,"drs_written":1,"lbm_read_time_us":278,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":322,"thread_start_us":60041,"threads_started":1}
I20260808 01:50:20.859572 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.011082
I20260808 01:50:20.860464 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.241s user 0.131s sys 0.009s Metrics: {"bytes_written":15871,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":1061,"drs_written":1,"lbm_read_time_us":269,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":429}
I20260808 01:50:20.867231 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=1.011839
I20260808 01:50:21.545300 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.995s user 0.179s sys 0.079s Metrics: {"bytes_written":18361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9796,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28882,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":396387,"dirs.run_cpu_time_us":2612,"dirs.run_wall_time_us":22457,"drs_written":1,"lbm_read_time_us":19927,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":48,"lbm_write_time_us":7463,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":208,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":525,"thread_start_us":587955,"threads_started":5}
I20260808 01:50:21.554215 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.121360
I20260808 01:50:21.866115 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.004s user 0.165s sys 0.037s Metrics: {"bytes_written":19964,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10909,"cfile_cache_miss":26,"cfile_cache_miss_bytes":32335,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":173630,"dirs.run_cpu_time_us":2469,"dirs.run_wall_time_us":39702,"drs_written":1,"lbm_read_time_us":3835,"lbm_reads_lt_1ms":50,"lbm_write_time_us":29959,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":19,"mutex_wait_us":68,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":586,"thread_start_us":248556,"threads_started":6}
I20260808 01:50:21.872641 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.060569
W20260808 01:50:22.036582 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 70157.6us
W20260808 01:50:22.049284 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 65980.3us
W20260808 01:50:22.283337 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 239416us
I20260808 01:50:22.338112 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.465s user 0.144s sys 0.001s Metrics: {"bytes_written":17146,"cfile_init":1,"dirs.queue_time_us":10214,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":263,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":480}
I20260808 01:50:22.364864 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.991570
I20260808 01:50:22.380646 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.825s user 0.184s sys 0.007s Metrics: {"bytes_written":19738,"cfile_init":1,"dirs.queue_time_us":431,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":4184,"drs_written":1,"lbm_read_time_us":256,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2928,"lbm_writes_lt_1ms":23,"mutex_wait_us":23684,"peak_mem_usage":0,"rows_written":578}
I20260808 01:50:22.434682 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.121360
I20260808 01:50:22.445304 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 1.576s user 0.163s sys 0.036s Metrics: {"bytes_written":18332,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28827,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":119919,"dirs.run_cpu_time_us":2197,"dirs.run_wall_time_us":16501,"drs_written":1,"lbm_read_time_us":98179,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":14664,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":524,"thread_start_us":139649,"threads_started":5}
I20260808 01:50:22.467989 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.121360
W20260808 01:50:22.689697 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 61269.4us
W20260808 01:50:22.732014 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.001s sys 0.000s
W20260808 01:50:22.777410 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 51785.5us
I20260808 01:50:22.862041 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.496s user 0.177s sys 0.004s Metrics: {"bytes_written":19186,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10358,"cfile_cache_miss":26,"cfile_cache_miss_bytes":30422,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":70110,"dirs.run_cpu_time_us":2115,"dirs.run_wall_time_us":28278,"drs_written":1,"lbm_read_time_us":16257,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":12015,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"mutex_wait_us":11066,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":556,"spinlock_wait_cycles":62720,"thread_start_us":113628,"threads_started":4}
I20260808 01:50:22.870723 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 5735 bytes on disk
I20260808 01:50:22.872551 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":310,"lbm_reads_lt_1ms":4}
I20260808 01:50:22.889886 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 5499 bytes on disk
I20260808 01:50:22.892434 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":328,"lbm_reads_lt_1ms":4}
I20260808 01:50:22.899825 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.060569
W20260808 01:50:22.955695 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 44566.9us
W20260808 01:50:23.046269 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 56679.9us
I20260808 01:50:23.424505 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.956s user 0.189s sys 0.011s Metrics: {"bytes_written":19582,"cfile_init":1,"dirs.queue_time_us":40700,"dirs.run_cpu_time_us":343,"dirs.run_wall_time_us":4052,"drs_written":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":571,"thread_start_us":42082,"threads_started":1}
I20260808 01:50:23.470409 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.060569
W20260808 01:50:22.757745 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.001s sys 0.000s
W20260808 01:50:23.718873 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.109s user 0.004s sys 0.000s
I20260808 01:50:23.753408 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 1.317s user 0.233s sys 0.028s Metrics: {"bytes_written":21259,"cfile_init":1,"dirs.queue_time_us":75717,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":3603,"drs_written":1,"lbm_read_time_us":285,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7550,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":636,"thread_start_us":75407,"threads_started":1}
I20260808 01:50:23.765944 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 6192 bytes on disk
I20260808 01:50:23.768260 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":323,"lbm_reads_lt_1ms":4}
I20260808 01:50:23.774330 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 5697 bytes on disk
I20260808 01:50:23.778843 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4}
W20260808 01:50:23.820467 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 60234.7us
I20260808 01:50:23.860508 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.121360
W20260808 01:50:24.246124 5270 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 80881.8us
W20260808 01:50:24.336552 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.186s user 0.001s sys 0.000s
W20260808 01:50:24.441859 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 62252.8us
W20260808 01:50:24.490386 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.105s user 0.004s sys 0.000s
I20260808 01:50:24.714988 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 1.814s user 0.166s sys 0.026s Metrics: {"bytes_written":17894,"cfile_init":1,"dirs.queue_time_us":168779,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":5186,"drs_written":1,"lbm_read_time_us":270,"lbm_reads_lt_1ms":4,"lbm_write_time_us":37488,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":505,"thread_start_us":169006,"threads_started":1}
I20260808 01:50:24.729597 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 4992 bytes on disk
I20260808 01:50:24.731478 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":306,"lbm_reads_lt_1ms":4}
I20260808 01:50:24.818197 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.844s user 0.202s sys 0.010s Metrics: {"bytes_written":20826,"cfile_init":1,"dirs.queue_time_us":39387,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":3883,"drs_written":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2643,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":620,"thread_start_us":34631,"threads_started":1}
I20260808 01:50:24.830926 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.121360
I20260808 01:50:25.006456 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=1.002972
W20260808 01:50:25.012930 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 77471.6us
W20260808 01:50:25.109378 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 67672.5us
I20260808 01:50:25.159262 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.687s user 0.130s sys 0.000s Metrics: {"bytes_written":14222,"cfile_init":1,"dirs.queue_time_us":56438,"dirs.run_cpu_time_us":326,"dirs.run_wall_time_us":20014,"drs_written":1,"lbm_read_time_us":334,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":365,"thread_start_us":56665,"threads_started":1}
I20260808 01:50:25.172924 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 5649 bytes on disk
I20260808 01:50:25.177809 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4}
I20260808 01:50:25.191896 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 5180 bytes on disk
I20260808 01:50:25.195951 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":355,"lbm_reads_lt_1ms":4}
W20260808 01:50:25.179128 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 51012.7us
I20260808 01:50:25.235199 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.013856
W20260808 01:50:25.314278 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 48541.1us
I20260808 01:50:25.412259 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.579s user 0.201s sys 0.004s Metrics: {"bytes_written":20836,"cfile_init":1,"dirs.queue_time_us":13290,"dirs.run_cpu_time_us":344,"dirs.run_wall_time_us":9187,"drs_written":1,"lbm_read_time_us":295,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11599,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":31,"peak_mem_usage":0,"rows_written":619,"thread_start_us":13495,"threads_started":1}
I20260808 01:50:25.438210 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.060569
W20260808 01:50:25.753243 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 73741.7us
W20260808 01:50:25.802769 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 69960.9us
W20260808 01:50:25.908536 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 102515us
W20260808 01:50:25.971446 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 58990.6us
W20260808 01:50:26.067240 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 94805.7us
W20260808 01:50:26.101423 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 106550us
I20260808 01:50:26.195816 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.954s user 0.234s sys 0.056s Metrics: {"bytes_written":21463,"cfile_cache_hit":10,"cfile_cache_hit_bytes":11987,"cfile_cache_miss":26,"cfile_cache_miss_bytes":35433,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":296441,"dirs.run_cpu_time_us":2501,"dirs.run_wall_time_us":36496,"drs_written":1,"lbm_read_time_us":3705,"lbm_reads_lt_1ms":50,"lbm_write_time_us":5470,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":645,"spinlock_wait_cycles":17726208,"thread_start_us":329522,"threads_started":6}
I20260808 01:50:26.206108 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.060569
W20260808 01:50:26.276027 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45710us
W20260808 01:50:26.350100 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 73134.1us
W20260808 01:50:26.418678 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 64398.6us
I20260808 01:50:26.486167 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.279s user 0.119s sys 0.003s Metrics: {"bytes_written":14172,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1512,"drs_written":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":363}
I20260808 01:50:26.493307 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.011967
W20260808 01:50:26.545301 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 72134.5us
W20260808 01:50:26.737346 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.091s user 0.001s sys 0.000s
W20260808 01:50:26.759660 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.062s user 0.004s sys 0.000s
W20260808 01:50:26.997741 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 98639.8us
W20260808 01:50:27.027105 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 46329.3us
W20260808 01:50:27.034188 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 48137.6us
W20260808 01:50:27.158699 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 61179.9us
W20260808 01:50:27.177043 5270 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 66034.2us
I20260808 01:50:27.267601 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 1.826s user 0.168s sys 0.000s Metrics: {"bytes_written":16325,"cfile_init":1,"compiler_manager_pool.queue_time_us":2684700,"compiler_manager_pool.run_cpu_time_us":59,"compiler_manager_pool.run_wall_time_us":59,"dirs.queue_time_us":11505,"dirs.run_cpu_time_us":350,"dirs.run_wall_time_us":10028,"drs_written":1,"lbm_read_time_us":346,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":447,"thread_start_us":11672,"threads_started":1}
I20260808 01:50:27.278931 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 6105 bytes on disk
I20260808 01:50:27.340507 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.846s user 0.312s sys 0.191s Metrics: {"bytes_written":23465,"cfile_cache_hit":10,"cfile_cache_hit_bytes":13408,"cfile_cache_miss":26,"cfile_cache_miss_bytes":39466,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":42359,"dirs.run_cpu_time_us":2256,"dirs.run_wall_time_us":22655,"drs_written":1,"lbm_read_time_us":3692,"lbm_reads_lt_1ms":50,"lbm_write_time_us":24928,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":723,"thread_start_us":421541,"threads_started":5}
I20260808 01:50:27.347575 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 7033 bytes on disk
I20260808 01:50:27.349498 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":283,"lbm_reads_lt_1ms":4}
I20260808 01:50:27.356611 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 6321 bytes on disk
I20260808 01:50:27.357674 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.036s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4}
I20260808 01:50:27.358374 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":297,"lbm_reads_lt_1ms":4}
I20260808 01:50:27.384939 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=1.012437
I20260808 01:50:27.399782 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 5092 bytes on disk
I20260808 01:50:27.401669 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
W20260808 01:50:27.468927 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45265.6us
I20260808 01:50:27.548589 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.060569
W20260808 01:50:27.562621 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 105679us
I20260808 01:50:27.786161 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.155s user 0.100s sys 0.000s Metrics: {"bytes_written":12374,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":324,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":294}
I20260808 01:50:27.794378 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=1.014566
W20260808 01:50:28.168627 5366 log.cc:927] Time spent T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356: Append to log took a long time: real 0.058s user 0.005s sys 0.000s
I20260808 01:50:28.260262 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.465s user 0.188s sys 0.002s Metrics: {"bytes_written":22546,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12773,"cfile_cache_miss":26,"cfile_cache_miss_bytes":37817,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":125958,"dirs.run_cpu_time_us":2336,"dirs.run_wall_time_us":10791,"drs_written":1,"lbm_read_time_us":4118,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2706,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":688,"thread_start_us":148809,"threads_started":3}
I20260808 01:50:28.303287 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 6738 bytes on disk
I20260808 01:50:28.315726 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":326,"lbm_reads_lt_1ms":4}
I20260808 01:50:28.324153 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
W20260808 01:50:28.364773 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.003s sys 0.000s
W20260808 01:50:28.422379 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.001s sys 0.000s
W20260808 01:50:28.472204 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.002s sys 0.000s
I20260808 01:50:28.506913 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 3.499s user 0.172s sys 0.006s Metrics: {"bytes_written":19863,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10874,"cfile_cache_miss":26,"cfile_cache_miss_bytes":32078,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":37978,"dirs.run_cpu_time_us":2335,"dirs.run_wall_time_us":13718,"drs_written":1,"lbm_read_time_us":89871,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2805,"lbm_writes_lt_1ms":23,"mutex_wait_us":2071,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":584,"spinlock_wait_cycles":55552,"thread_start_us":257586,"threads_started":4}
I20260808 01:50:28.515529 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 5759 bytes on disk
W20260808 01:50:28.547688 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 106057us
I20260808 01:50:28.570971 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":350,"lbm_reads_lt_1ms":4}
I20260808 01:50:28.579249 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.121360
W20260808 01:50:28.667972 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 47686.6us
W20260808 01:50:28.709702 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 79894us
W20260808 01:50:28.856442 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.004s sys 0.000s
I20260808 01:50:28.874104 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 1.488s user 0.184s sys 0.012s Metrics: {"bytes_written":22020,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12368,"cfile_cache_miss":26,"cfile_cache_miss_bytes":36462,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":191249,"dirs.run_cpu_time_us":3158,"dirs.run_wall_time_us":77195,"drs_written":1,"lbm_read_time_us":55208,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":46,"lbm_write_time_us":30170,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"mutex_wait_us":10761,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":666,"spinlock_wait_cycles":2432,"thread_start_us":313300,"threads_started":7}
I20260808 01:50:28.882740 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.121360
W20260808 01:50:29.050500 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 52654.1us
W20260808 01:50:29.225682 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 107138us
I20260808 01:50:29.233546 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.908s user 0.104s sys 0.004s Metrics: {"bytes_written":12330,"cfile_init":1,"dirs.queue_time_us":8777,"dirs.run_cpu_time_us":367,"dirs.run_wall_time_us":8745,"drs_written":1,"lbm_read_time_us":307,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17429,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":292,"thread_start_us":17666,"threads_started":1}
I20260808 01:50:29.249037 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 2942 bytes on disk
I20260808 01:50:29.257345 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":269,"lbm_reads_lt_1ms":4}
I20260808 01:50:29.273911 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.014214
I20260808 01:50:29.382534 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.803s user 0.182s sys 0.000s Metrics: {"bytes_written":19814,"cfile_init":1,"dirs.queue_time_us":26267,"dirs.run_cpu_time_us":329,"dirs.run_wall_time_us":3002,"drs_written":1,"lbm_read_time_us":271,"lbm_reads_lt_1ms":4,"lbm_write_time_us":30676,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":582,"thread_start_us":26313,"threads_started":1}
I20260808 01:50:29.391868 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 5724 bytes on disk
I20260808 01:50:29.393828 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":306,"lbm_reads_lt_1ms":4}
I20260808 01:50:29.401158 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=1.010398
W20260808 01:50:29.411211 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 118098us
W20260808 01:50:29.669756 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 125170us
W20260808 01:50:29.874100 5367 log.cc:927] Time spent T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356: Append to log took a long time: real 0.059s user 0.000s sys 0.010s
W20260808 01:50:29.906980 5356 log.cc:927] Time spent T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Append to log took a long time: real 0.104s user 0.001s sys 0.003s
I20260808 01:50:30.103272 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 1.220s user 0.162s sys 0.033s Metrics: {"bytes_written":19939,"cfile_init":1,"dirs.queue_time_us":6945,"dirs.run_cpu_time_us":531,"dirs.run_wall_time_us":5448,"drs_written":1,"lbm_read_time_us":312,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9908,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":585,"thread_start_us":6972,"threads_started":1}
I20260808 01:50:30.111904 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 6544 bytes on disk
I20260808 01:50:30.115046 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":429,"lbm_reads_lt_1ms":4}
W20260808 01:50:30.118615 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 44990.3us
I20260808 01:50:30.123090 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 6089 bytes on disk
I20260808 01:50:30.127328 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":382,"lbm_reads_lt_1ms":4}
I20260808 01:50:30.177428 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 5731 bytes on disk
W20260808 01:50:30.227517 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 51907.6us
I20260808 01:50:30.279713 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":391,"lbm_reads_lt_1ms":4}
I20260808 01:50:30.439054 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.010623
W20260808 01:50:30.640403 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.231s user 0.004s sys 0.000s
W20260808 01:50:30.782269 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 58324.2us
W20260808 01:50:30.844190 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 63367.2us
I20260808 01:50:31.003662 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.727s user 0.264s sys 0.099s Metrics: {"bytes_written":25939,"cfile_cache_hit":10,"cfile_cache_hit_bytes":15126,"cfile_cache_miss":26,"cfile_cache_miss_bytes":44406,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":302615,"dirs.run_cpu_time_us":2343,"dirs.run_wall_time_us":19960,"drs_written":1,"lbm_read_time_us":3982,"lbm_reads_lt_1ms":46,"lbm_write_time_us":96163,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":19,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":817,"thread_start_us":359454,"threads_started":6}
I20260808 01:50:31.023860 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.121360
W20260808 01:50:31.045002 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 82136.8us
W20260808 01:50:31.103439 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 57718.8us
W20260808 01:50:31.147540 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 43582.4us
W20260808 01:50:31.271952 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 56733.7us
I20260808 01:50:31.309332 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.879s user 0.194s sys 0.001s Metrics: {"bytes_written":24127,"cfile_cache_hit":10,"cfile_cache_hit_bytes":13850,"cfile_cache_miss":26,"cfile_cache_miss_bytes":40650,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":100212,"dirs.run_cpu_time_us":2357,"dirs.run_wall_time_us":46421,"drs_written":1,"lbm_read_time_us":69584,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2766,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":747,"spinlock_wait_cycles":1920,"thread_start_us":156523,"threads_started":6}
I20260808 01:50:31.318603 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 7282 bytes on disk
I20260808 01:50:31.320536 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":324,"lbm_reads_lt_1ms":4}
I20260808 01:50:31.326695 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.060569
W20260808 01:50:31.518499 5375 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.004s sys 0.000s
I20260808 01:50:31.758538 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.381s user 0.126s sys 0.000s Metrics: {"bytes_written":13625,"cfile_init":1,"dirs.queue_time_us":101,"dirs.run_cpu_time_us":282,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":294,"lbm_reads_lt_1ms":4,"lbm_write_time_us":55167,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":344}
I20260808 01:50:31.769259 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.012661
W20260808 01:50:31.867229 5256 debug-util.cc:398] Leaking SignalData structure 0x7b080006a860 after lost signal to thread 5127
W20260808 01:50:31.868582 5256 debug-util.cc:398] Leaking SignalData structure 0x7b080007eb40 after lost signal to thread 5259
W20260808 01:50:31.966892 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 143822us
W20260808 01:50:31.975111 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 154491us
W20260808 01:50:32.056002 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 51642.5us
I20260808 01:50:32.239660 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 1.800s user 0.200s sys 0.008s Metrics: {"bytes_written":25282,"cfile_cache_hit":10,"cfile_cache_hit_bytes":14687,"cfile_cache_miss":26,"cfile_cache_miss_bytes":43129,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":42200,"dirs.run_cpu_time_us":2024,"dirs.run_wall_time_us":23482,"drs_written":1,"lbm_read_time_us":3534,"lbm_reads_lt_1ms":46,"lbm_write_time_us":7988,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":793,"thread_start_us":40796,"threads_started":4}
I20260808 01:50:32.249537 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 7715 bytes on disk
I20260808 01:50:32.251322 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4}
I20260808 01:50:32.257472 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.121360
W20260808 01:50:32.577889 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 55237.9us
I20260808 01:50:32.803936 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.779s user 0.203s sys 0.004s Metrics: {"bytes_written":20898,"cfile_init":1,"dirs.queue_time_us":10694,"dirs.run_cpu_time_us":370,"dirs.run_wall_time_us":10745,"drs_written":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4,"lbm_write_time_us":45661,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":20,"peak_mem_usage":0,"rows_written":623,"thread_start_us":13247,"threads_started":1}
I20260808 01:50:32.925467 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=1.010469
W20260808 01:50:33.100780 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.075s user 0.004s sys 0.000s
W20260808 01:50:33.175019 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 62136.4us
W20260808 01:50:33.203455 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 60499.3us
W20260808 01:50:33.313477 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 58596.9us
W20260808 01:50:33.330428 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 50900us
I20260808 01:50:33.476262 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 1.218s user 0.205s sys 0.000s Metrics: {"bytes_written":21915,"cfile_init":1,"dirs.queue_time_us":20955,"dirs.run_cpu_time_us":335,"dirs.run_wall_time_us":5024,"drs_written":1,"lbm_read_time_us":362,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2666,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":663,"thread_start_us":26140,"threads_started":1}
I20260808 01:50:33.524626 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=1.006960
I20260808 01:50:33.580690 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.809s user 0.207s sys 0.007s Metrics: {"bytes_written":26307,"cfile_cache_hit":10,"cfile_cache_hit_bytes":15418,"cfile_cache_miss":26,"cfile_cache_miss_bytes":45316,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":109115,"dirs.run_cpu_time_us":1684,"dirs.run_wall_time_us":17287,"drs_written":1,"lbm_read_time_us":3816,"lbm_reads_lt_1ms":50,"lbm_write_time_us":44319,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":833,"thread_start_us":121586,"threads_started":5}
I20260808 01:50:33.588634 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 8147 bytes on disk
I20260808 01:50:33.858386 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.269s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":297,"lbm_reads_lt_1ms":4}
I20260808 01:50:34.101785 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
W20260808 01:50:34.277778 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 54605.2us
W20260808 01:50:34.365979 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 58856.1us
I20260808 01:50:34.403134 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 1.476s user 0.213s sys 0.000s Metrics: {"bytes_written":25848,"cfile_cache_hit":10,"cfile_cache_hit_bytes":15086,"cfile_cache_miss":26,"cfile_cache_miss_bytes":44228,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":5717,"dirs.run_cpu_time_us":339,"dirs.run_wall_time_us":16441,"drs_written":1,"lbm_read_time_us":47887,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":47,"lbm_write_time_us":2700,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":815,"spinlock_wait_cycles":28229120,"thread_start_us":6014,"threads_started":1}
I20260808 01:50:34.417470 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.121360
W20260808 01:50:34.706532 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 63050.8us
W20260808 01:50:34.744727 5372 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.004s sys 0.000s
I20260808 01:50:35.134974 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.032s user 0.129s sys 0.012s Metrics: {"bytes_written":15396,"cfile_init":1,"dirs.queue_time_us":88756,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":1892,"drs_written":1,"lbm_read_time_us":417,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21575,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":410,"thread_start_us":91054,"threads_started":1}
I20260808 01:50:35.170083 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 4041 bytes on disk
I20260808 01:50:35.173790 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":355,"lbm_reads_lt_1ms":4}
I20260808 01:50:35.179718 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.013109
W20260808 01:50:35.434875 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 93935us
W20260808 01:50:35.435937 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 85022.3us
W20260808 01:50:35.462828 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 149554us
W20260808 01:50:35.966709 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 122103us
W20260808 01:50:36.018215 5008 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 50183.8us
W20260808 01:50:36.060940 5368 log.cc:927] Time spent T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.054s user 0.007s sys 0.005s
W20260808 01:50:36.062191 5379 log.cc:927] Time spent T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.057s user 0.008s sys 0.004s
W20260808 01:50:36.064929 5378 log.cc:927] Time spent T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.051s user 0.004s sys 0.008s
W20260808 01:50:36.101953 5353 log.cc:927] Time spent T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.092s user 0.006s sys 0.005s
I20260808 01:50:36.350085 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 2.825s user 0.220s sys 0.000s Metrics: {"bytes_written":26399,"cfile_cache_hit":10,"cfile_cache_hit_bytes":15490,"cfile_cache_miss":26,"cfile_cache_miss_bytes":45424,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":133087,"dirs.run_cpu_time_us":1955,"dirs.run_wall_time_us":25077,"drs_written":1,"lbm_read_time_us":21037,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":18024,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":837,"thread_start_us":120635,"threads_started":7}
I20260808 01:50:36.368443 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 8164 bytes on disk
I20260808 01:50:36.370342 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":337,"lbm_reads_lt_1ms":4}
I20260808 01:50:36.372438 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 1.954s user 0.194s sys 0.009s Metrics: {"bytes_written":21695,"cfile_init":1,"dirs.queue_time_us":8107,"dirs.run_cpu_time_us":309,"dirs.run_wall_time_us":1222,"drs_written":1,"lbm_read_time_us":290,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8974,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":655,"thread_start_us":24570,"threads_started":1}
I20260808 01:50:36.381377 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2): perf score=1.012282
I20260808 01:50:36.384008 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.121360
I20260808 01:50:36.405344 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 1.224s user 0.362s sys 0.131s Metrics: {"bytes_written":30570,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53834,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":309144,"dirs.run_cpu_time_us":2618,"dirs.run_wall_time_us":49274,"drs_written":1,"lbm_read_time_us":57388,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":43,"lbm_write_time_us":20245,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":996,"spinlock_wait_cycles":9088,"thread_start_us":347042,"threads_started":6}
I20260808 01:50:36.413247 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 9609 bytes on disk
I20260808 01:50:36.415385 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":285,"lbm_reads_lt_1ms":4}
I20260808 01:50:36.434492 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.121360
W20260808 01:50:36.621752 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 56129.5us
W20260808 01:50:36.681471 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 58905.2us
I20260808 01:50:36.733955 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.347s user 0.178s sys 0.000s Metrics: {"bytes_written":20436,"cfile_init":1,"dirs.queue_time_us":98,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":267,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11848,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":608}
I20260808 01:50:36.740859 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 5956 bytes on disk
I20260808 01:50:36.759490 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":425,"lbm_reads_lt_1ms":4}
I20260808 01:50:36.803303 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 5155 bytes on disk
I20260808 01:50:36.806938 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":974,"lbm_reads_lt_1ms":4}
I20260808 01:50:36.813056 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 3158 bytes on disk
I20260808 01:50:36.894315 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":434,"lbm_reads_lt_1ms":4}
W20260808 01:50:36.916450 5270 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.29.180.65:33111, user_credentials={real_user=slave}} blocked reactor thread for 89187.6us
W20260808 01:50:36.919258 5269 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 81037.3us
I20260808 01:50:36.935434 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.013814
I20260808 01:50:36.938122 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.499s user 0.172s sys 0.004s Metrics: {"bytes_written":20897,"cfile_init":1,"dirs.queue_time_us":6061,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":1485,"drs_written":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10750,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":624}
I20260808 01:50:36.945072 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 6118 bytes on disk
I20260808 01:50:36.947355 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":307,"lbm_reads_lt_1ms":4}
I20260808 01:50:36.953421 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=1.013158
W20260808 01:50:37.098869 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 47136.2us
W20260808 01:50:37.245711 5374 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.003s sys 0.000s
W20260808 01:50:37.421587 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 49919.8us
I20260808 01:50:37.450484 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.495s user 0.243s sys 0.043s Metrics: {"bytes_written":29571,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17698,"cfile_cache_miss":26,"cfile_cache_miss_bytes":51842,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":127697,"dirs.run_cpu_time_us":2202,"dirs.run_wall_time_us":12342,"drs_written":1,"lbm_read_time_us":3606,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2962,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":958,"thread_start_us":123717,"threads_started":4}
I20260808 01:50:37.458743 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.121360
W20260808 01:50:37.460318 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.004s sys 0.000s
W20260808 01:50:37.589701 5370 meta_cache.cc:1261] Time spent looking up entry by key: real 0.124s user 0.004s sys 0.000s
I20260808 01:50:37.853904 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.394s user 0.228s sys 0.001s Metrics: {"bytes_written":22272,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":308,"dirs.run_wall_time_us":966,"drs_written":1,"lbm_read_time_us":333,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3146,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":675}
I20260808 01:50:37.862903 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 9343 bytes on disk
I20260808 01:50:37.866707 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":351,"lbm_reads_lt_1ms":4}
I20260808 01:50:37.873520 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 6583 bytes on disk
I20260808 01:50:37.875713 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":353,"lbm_reads_lt_1ms":4}
I20260808 01:50:37.885986 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.010775
W20260808 01:50:37.913779 4979 debug-util.cc:398] Leaking SignalData structure 0x7b08000f12e0 after lost signal to thread 4848
W20260808 01:50:37.915117 4979 debug-util.cc:398] Leaking SignalData structure 0x7b0800100ec0 after lost signal to thread 4982
W20260808 01:50:37.941452 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 59895.6us
W20260808 01:50:38.048403 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 71398.2us
I20260808 01:50:38.072947 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 1.137s user 0.280s sys 0.084s Metrics: {"bytes_written":31213,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18827,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54909,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":365728,"dirs.run_cpu_time_us":2851,"dirs.run_wall_time_us":17018,"drs_written":1,"lbm_read_time_us":16798,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2840,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1020,"spinlock_wait_cycles":20864,"thread_start_us":489254,"threads_started":5}
I20260808 01:50:38.079978 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.121360
I20260808 01:50:38.184289 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.68:0
--local_ip_for_outbound_sockets=127.29.180.68
--webserver_interface=127.29.180.68
--webserver_port=0
--tserver_master_addrs=127.29.180.126:32885
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1 with env {}
W20260808 01:50:38.226203 4869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 55393.6us
I20260808 01:50:38.266590 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.380s user 0.277s sys 0.044s Metrics: {"bytes_written":32143,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19523,"cfile_cache_miss":26,"cfile_cache_miss_bytes":57123,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":31853,"dirs.run_cpu_time_us":2516,"dirs.run_wall_time_us":9104,"drs_written":1,"lbm_read_time_us":3891,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2987,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1058,"thread_start_us":54585,"threads_started":4}
I20260808 01:50:38.273994 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.121360
I20260808 01:50:38.443500 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.363s user 0.172s sys 0.002s Metrics: {"bytes_written":19941,"cfile_init":1,"dirs.queue_time_us":95,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":665,"drs_written":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":587}
I20260808 01:50:38.462656 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 9917 bytes on disk
I20260808 01:50:38.464473 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":297,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.469873 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 5723 bytes on disk
I20260808 01:50:38.480194 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.205s user 0.194s sys 0.005s Metrics: {"bytes_written":21330,"cfile_init":1,"dirs.queue_time_us":111,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1540,"drs_written":1,"lbm_read_time_us":281,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2682,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":639}
I20260808 01:50:38.483304 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":318,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.487679 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 10274 bytes on disk
I20260808 01:50:38.488791 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.012035
I20260808 01:50:38.490628 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.495873 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 6302 bytes on disk
I20260808 01:50:38.497716 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":292,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.503669 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05): perf score=1.013739
I20260808 01:50:38.697974 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 2.204s user 0.244s sys 0.008s Metrics: {"bytes_written":29234,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17462,"cfile_cache_miss":26,"cfile_cache_miss_bytes":51200,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":51551,"dirs.run_cpu_time_us":2947,"dirs.run_wall_time_us":18697,"drs_written":1,"lbm_read_time_us":12500,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":154227,"lbm_writes_1-10_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":21,"mutex_wait_us":334,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":945,"spinlock_wait_cycles":81408,"thread_start_us":110237,"threads_started":5}
I20260808 01:50:38.705315 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 9186 bytes on disk
I20260808 01:50:38.707118 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":365,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.712888 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.121360
I20260808 01:50:38.858269 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.354s user 0.290s sys 0.038s Metrics: {"bytes_written":29653,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17810,"cfile_cache_miss":26,"cfile_cache_miss_bytes":52354,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":9883,"dirs.run_cpu_time_us":2638,"dirs.run_wall_time_us":9923,"drs_written":1,"lbm_read_time_us":3524,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2700,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":964,"thread_start_us":88621,"threads_started":3}
I20260808 01:50:38.865340 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 9425 bytes on disk
I20260808 01:50:38.867313 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":336,"lbm_reads_lt_1ms":4}
I20260808 01:50:38.872893 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.121360
I20260808 01:50:39.112545 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.239s user 0.203s sys 0.020s Metrics: {"bytes_written":22498,"cfile_init":1,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":2946,"drs_written":1,"lbm_read_time_us":299,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":682,"thread_start_us":21464,"threads_started":1,"wal-append.queue_time_us":21884}
I20260808 01:50:39.119200 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 6676 bytes on disk
I20260808 01:50:39.127009 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4}
I20260808 01:50:39.132588 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2): perf score=1.012443
I20260808 01:50:39.297458 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.571s user 0.222s sys 0.008s Metrics: {"bytes_written":22243,"cfile_init":1,"dirs.queue_time_us":17277,"dirs.run_cpu_time_us":315,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":674,"thread_start_us":25530,"threads_started":2,"wal-append.queue_time_us":8315}
I20260808 01:50:39.304651 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05): perf score=1.013807
I20260808 01:50:39.343780 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.852s user 0.273s sys 0.038s Metrics: {"bytes_written":33416,"cfile_cache_hit":10,"cfile_cache_hit_bytes":20511,"cfile_cache_miss":26,"cfile_cache_miss_bytes":59805,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":63790,"dirs.run_cpu_time_us":2503,"dirs.run_wall_time_us":9139,"drs_written":1,"lbm_read_time_us":12815,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":44,"lbm_write_time_us":2783,"lbm_writes_lt_1ms":23,"mutex_wait_us":10265,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1112,"thread_start_us":86760,"threads_started":5}
I20260808 01:50:39.351009 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.121360
I20260808 01:50:39.873137 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.519s user 0.210s sys 0.008s Metrics: {"bytes_written":22505,"cfile_init":1,"dirs.queue_time_us":9851,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":304,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3718,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":683,"thread_start_us":12920,"threads_started":1,"wal-append.queue_time_us":16872}
I20260808 01:50:39.894049 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 10729 bytes on disk
I20260808 01:50:39.896194 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":332,"lbm_reads_lt_1ms":4}
I20260808 01:50:39.902288 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 6691 bytes on disk
I20260808 01:50:39.917081 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":333,"lbm_reads_lt_1ms":4}
I20260808 01:50:39.930531 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=1.013077
W20260808 01:50:39.962850 6340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:50:39.964051 6340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:50:39.965358 6340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260808 01:50:40.010316 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.877s user 0.274s sys 0.055s Metrics: {"bytes_written":31899,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19416,"cfile_cache_miss":26,"cfile_cache_miss_bytes":56980,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":38273,"dirs.run_cpu_time_us":2424,"dirs.run_wall_time_us":14436,"drs_written":1,"lbm_read_time_us":17258,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":44,"lbm_write_time_us":23519,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1052,"thread_start_us":100993,"threads_started":5}
I20260808 01:50:40.017159 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 10221 bytes on disk
I20260808 01:50:40.018916 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":268,"lbm_reads_lt_1ms":4}
I20260808 01:50:40.024170 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.121360
I20260808 01:50:40.078667 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.773s user 0.254s sys 0.014s Metrics: {"bytes_written":29653,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17810,"cfile_cache_miss":26,"cfile_cache_miss_bytes":52428,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":48052,"dirs.run_cpu_time_us":2685,"dirs.run_wall_time_us":22555,"drs_written":1,"lbm_read_time_us":15632,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":18180,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"mutex_wait_us":9,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":964,"spinlock_wait_cycles":31104,"thread_start_us":93212,"threads_started":5}
W20260808 01:50:40.082111 6340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:50:40.083060 6340 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260808 01:50:40.084501 6340 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260808 01:50:40.085433 6340 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.68
I20260808 01:50:40.086403 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 9425 bytes on disk
I20260808 01:50:40.088147 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4}
I20260808 01:50:40.093954 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 7959 bytes on disk
I20260808 01:50:40.095741 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":279,"lbm_reads_lt_1ms":4}
I20260808 01:50:40.104005 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.060569
I20260808 01:50:40.162315 6340 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
--webserver_interface=127.29.180.68
--webserver_port=0
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.29.180.126:32885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:50:40.164516 6340 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:50:40.167308 6340 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:50:40.243269 6388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:50:40.369515 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.335s user 0.183s sys 0.003s Metrics: {"bytes_written":20624,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":386,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":612,"thread_start_us":11365,"threads_started":1,"wal-append.queue_time_us":11010}
I20260808 01:50:40.389712 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e): perf score=1.013348
I20260808 01:50:40.428764 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.307s user 0.141s sys 0.003s Metrics: {"bytes_written":15218,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":388,"dirs.run_wall_time_us":1629,"drs_written":1,"lbm_read_time_us":293,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9973,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":403,"thread_start_us":9796,"threads_started":1,"wal-append.queue_time_us":9825}
I20260808 01:50:40.444666 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.011693
I20260808 01:50:40.586465 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.655s user 0.280s sys 0.008s Metrics: {"bytes_written":30515,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18338,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53906,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":42447,"dirs.run_cpu_time_us":2595,"dirs.run_wall_time_us":22303,"drs_written":1,"lbm_read_time_us":4782,"lbm_reads_lt_1ms":46,"lbm_write_time_us":3386,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":993,"thread_start_us":59063,"threads_started":4}
I20260808 01:50:40.593899 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.060569
I20260808 01:50:40.843452 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.453s user 0.281s sys 0.027s Metrics: {"bytes_written":29337,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17552,"cfile_cache_miss":26,"cfile_cache_miss_bytes":51714,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":43557,"dirs.run_cpu_time_us":2550,"dirs.run_wall_time_us":19921,"drs_written":1,"lbm_read_time_us":4270,"lbm_reads_lt_1ms":50,"lbm_write_time_us":5123,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":79,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":950,"spinlock_wait_cycles":82176,"thread_start_us":53277,"threads_started":4}
I20260808 01:50:40.851812 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.060569
I20260808 01:50:40.868165 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.273s user 0.128s sys 0.001s Metrics: {"bytes_written":13151,"cfile_init":1,"dirs.queue_time_us":9435,"dirs.run_cpu_time_us":419,"dirs.run_wall_time_us":1653,"drs_written":1,"lbm_read_time_us":311,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15132,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":324,"thread_start_us":6777,"threads_started":1,"wal-append.queue_time_us":4829}
I20260808 01:50:40.877790 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=1.012742
I20260808 01:50:41.256549 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.811s user 0.293s sys 0.010s Metrics: {"bytes_written":31747,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19343,"cfile_cache_miss":26,"cfile_cache_miss_bytes":56647,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":41075,"dirs.run_cpu_time_us":2715,"dirs.run_wall_time_us":19845,"drs_written":1,"lbm_read_time_us":12808,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":24500,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1048,"spinlock_wait_cycles":27392,"thread_start_us":47378,"threads_started":5}
I20260808 01:50:41.266567 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.060569
I20260808 01:50:41.468532 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.590s user 0.256s sys 0.002s Metrics: {"bytes_written":30293,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18282,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53668,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":21858,"dirs.run_cpu_time_us":1798,"dirs.run_wall_time_us":14437,"drs_written":1,"lbm_read_time_us":25918,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":2926,"lbm_writes_lt_1ms":23,"mutex_wait_us":5,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":990,"spinlock_wait_cycles":27648,"thread_start_us":25608,"threads_started":5}
I20260808 01:50:41.486210 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 9670 bytes on disk
I20260808 01:50:41.488303 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4}
I20260808 01:50:41.501024 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 9643 bytes on disk
I20260808 01:50:41.514127 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":326,"lbm_reads_lt_1ms":4}
I20260808 01:50:41.554558 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.060569
I20260808 01:50:41.590250 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.737s user 0.168s sys 0.009s Metrics: {"bytes_written":16342,"cfile_init":1,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":406,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3458,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":447,"thread_start_us":28698,"threads_started":1,"wal-append.queue_time_us":12631}
I20260808 01:50:41.619504 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 9282 bytes on disk
I20260808 01:50:41.622843 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":431,"lbm_reads_lt_1ms":4}
I20260808 01:50:41.628304 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=1.014187
I20260808 01:50:41.873880 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.606s user 0.117s sys 0.012s Metrics: {"bytes_written":12574,"cfile_init":1,"dirs.queue_time_us":108,"dirs.run_cpu_time_us":349,"dirs.run_wall_time_us":1485,"drs_written":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2785,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":302,"thread_start_us":7720,"threads_started":1,"wal-append.queue_time_us":7438}
I20260808 01:50:41.892604 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 10965 bytes on disk
I20260808 01:50:41.897480 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.004s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":790,"lbm_reads_lt_1ms":8}
I20260808 01:50:41.899230 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.327s user 0.154s sys 0.006s Metrics: {"bytes_written":16025,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":981,"drs_written":1,"lbm_read_time_us":296,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":434,"thread_start_us":16238,"threads_started":1,"wal-append.queue_time_us":15325}
I20260808 01:50:41.903388 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.010992
I20260808 01:50:41.905906 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=1.013756
I20260808 01:50:42.434265 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.528s user 0.264s sys 0.043s Metrics: {"bytes_written":30939,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18686,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54764,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":70346,"dirs.run_cpu_time_us":2899,"dirs.run_wall_time_us":23598,"drs_written":1,"lbm_read_time_us":4414,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2814,"lbm_writes_lt_1ms":23,"mutex_wait_us":445,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1012,"spinlock_wait_cycles":166144,"thread_start_us":81997,"threads_started":5}
I20260808 01:50:42.442163 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(86d58fdfa95b45f68c0648a585924d05): perf score=0.060569
I20260808 01:50:42.578293 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.133s user 0.123s sys 0.008s Metrics: {"bytes_written":13639,"cfile_init":1,"dirs.queue_time_us":96,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":1649,"drs_written":1,"lbm_read_time_us":366,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3053,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":344,"thread_start_us":8061,"threads_started":1,"wal-append.queue_time_us":7743}
I20260808 01:50:42.584923 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05): perf score=1.011998
I20260808 01:50:42.840620 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.936s user 0.336s sys 0.019s Metrics: {"bytes_written":33614,"cfile_cache_hit":10,"cfile_cache_hit_bytes":20640,"cfile_cache_miss":26,"cfile_cache_miss_bytes":60272,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":38415,"dirs.run_cpu_time_us":3084,"dirs.run_wall_time_us":26636,"drs_written":1,"lbm_read_time_us":4810,"lbm_reads_lt_1ms":46,"lbm_write_time_us":3727,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1119,"spinlock_wait_cycles":50560,"thread_start_us":39592,"threads_started":5}
I20260808 01:50:42.849293 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 10789 bytes on disk
I20260808 01:50:42.851773 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":407,"lbm_reads_lt_1ms":4}
I20260808 01:50:42.867268 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 10187 bytes on disk
I20260808 01:50:42.869565 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":328,"lbm_reads_lt_1ms":4}
I20260808 01:50:42.875494 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.060569
I20260808 01:50:42.892748 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 1.264s user 0.335s sys 0.046s Metrics: {"bytes_written":30727,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18518,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54278,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":49084,"dirs.run_cpu_time_us":2781,"dirs.run_wall_time_us":16293,"drs_written":1,"lbm_read_time_us":24629,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":48,"lbm_write_time_us":2756,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1003,"spinlock_wait_cycles":2432,"thread_start_us":141976,"threads_started":7,"wal-append.queue_time_us":36099}
I20260808 01:50:42.902328 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.060569
I20260808 01:50:43.230074 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.354s user 0.174s sys 0.001s Metrics: {"bytes_written":17106,"cfile_init":1,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":360,"dirs.run_wall_time_us":1554,"drs_written":1,"lbm_read_time_us":245,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11874,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":477,"thread_start_us":10369,"threads_started":1,"wal-append.queue_time_us":9918}
I20260808 01:50:43.252840 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=1.012939
I20260808 01:50:43.385011 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.481s user 0.179s sys 0.014s Metrics: {"bytes_written":15380,"cfile_init":1,"dirs.queue_time_us":110,"dirs.run_cpu_time_us":307,"dirs.run_wall_time_us":976,"drs_written":1,"lbm_read_time_us":9449,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":5057,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":409,"thread_start_us":9740,"threads_started":1,"wal-append.queue_time_us":9513}
I20260808 01:50:43.393720 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 9739 bytes on disk
I20260808 01:50:43.395715 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":283,"lbm_reads_lt_1ms":4}
I20260808 01:50:43.401597 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=1.009743
I20260808 01:50:43.434983 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.849s user 0.315s sys 0.022s Metrics: {"bytes_written":29653,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17805,"cfile_cache_miss":26,"cfile_cache_miss_bytes":52251,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":43176,"dirs.run_cpu_time_us":2454,"dirs.run_wall_time_us":7586,"drs_written":1,"lbm_read_time_us":17136,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":4290,"lbm_writes_lt_1ms":23,"mutex_wait_us":41,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":964,"spinlock_wait_cycles":10018560,"thread_start_us":68056,"threads_started":6}
I20260808 01:50:43.442006 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.060569
W20260808 01:50:41.615684 6386 debug-util.cc:398] Leaking SignalData structure 0x7b08000372c0 after lost signal to thread 6340
W20260808 01:50:43.718078 6386 kernel_stack_watchdog.cc:198] Thread 6340 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 379ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260808 01:50:40.245952 6387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:50:43.724571 6340 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 3.490s user 0.740s sys 1.418s
W20260808 01:50:43.725741 6340 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 3.491s user 0.740s sys 1.418s
I20260808 01:50:43.736017 6340 server_base.cc:1061] running on GCE node
W20260808 01:50:43.737432 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:50:43.739837 6340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:50:43.749467 6340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:50:43.753252 6340 hybrid_clock.cc:648] HybridClock initialized: now 1786153843753300 us; error 192 us; skew 500 ppm
I20260808 01:50:43.754810 6340 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:50:43.759514 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.357s user 0.236s sys 0.010s Metrics: {"bytes_written":30515,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18338,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53754,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":14599,"dirs.run_cpu_time_us":2520,"dirs.run_wall_time_us":6553,"drs_written":1,"lbm_read_time_us":3650,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2802,"lbm_writes_lt_1ms":23,"mutex_wait_us":367,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":993,"spinlock_wait_cycles":122496,"thread_start_us":33028,"threads_started":5}
I20260808 01:50:43.767691 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 9670 bytes on disk
I20260808 01:50:43.770190 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":443,"lbm_reads_lt_1ms":4}
I20260808 01:50:43.777495 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(0c44eefb43024e9290fdba8647e745fe): perf score=0.030173
I20260808 01:50:43.785506 6340 webserver.cc:533] Webserver started at http://127.29.180.68:38843/ using document root <none> and password file <none>
I20260808 01:50:43.787737 6340 fs_manager.cc:362] Metadata directory not provided
I20260808 01:50:43.788218 6340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:50:43.789134 6340 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:50:43.796602 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/instance:
uuid: "c37035739bf347d49e5ed4b2f884adac"
format_stamp: "Formatted at 2026-08-08 01:50:43 on dist-test-slave-p3t5"
I20260808 01:50:43.798686 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/instance:
uuid: "c37035739bf347d49e5ed4b2f884adac"
format_stamp: "Formatted at 2026-08-08 01:50:43 on dist-test-slave-p3t5"
I20260808 01:50:43.800184 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-2/instance:
uuid: "c37035739bf347d49e5ed4b2f884adac"
format_stamp: "Formatted at 2026-08-08 01:50:43 on dist-test-slave-p3t5"
I20260808 01:50:43.801589 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance:
uuid: "c37035739bf347d49e5ed4b2f884adac"
format_stamp: "Formatted at 2026-08-08 01:50:43 on dist-test-slave-p3t5"
I20260808 01:50:43.812899 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.370s user 0.151s sys 0.032s Metrics: {"bytes_written":15494,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":360,"dirs.run_wall_time_us":922,"drs_written":1,"lbm_read_time_us":402,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3372,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":416,"thread_start_us":23030,"threads_started":1,"wal-append.queue_time_us":23395}
I20260808 01:50:43.815855 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.562s user 0.252s sys 0.026s Metrics: {"bytes_written":30293,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18287,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53601,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":22752,"dirs.run_cpu_time_us":2474,"dirs.run_wall_time_us":18236,"drs_written":1,"lbm_read_time_us":8305,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":2866,"lbm_writes_lt_1ms":23,"mutex_wait_us":497,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":990,"spinlock_wait_cycles":15616,"thread_start_us":40031,"threads_started":4}
I20260808 01:50:43.822412 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.060569
I20260808 01:50:43.824038 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2): perf score=1.013408
I20260808 01:50:43.832609 6340 fs_manager.cc:696] Time spent creating directory manager: real 0.030s user 0.013s sys 0.009s
I20260808 01:50:43.894781 6465 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:43.895421 6465 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:43.909379 6466 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:43.913578 6340 fs_manager.cc:730] Time spent opening block manager: real 0.075s user 0.010s sys 0.049s
I20260808 01:50:43.914173 6340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal
uuid: "c37035739bf347d49e5ed4b2f884adac"
format_stamp: "Formatted at 2026-08-08 01:50:43 on dist-test-slave-p3t5"
I20260808 01:50:43.914896 6340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-2/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:50:43.968797 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.190s user 0.082s sys 0.004s Metrics: {"bytes_written":10280,"cfile_init":1,"dirs.queue_time_us":101,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":917,"drs_written":1,"lbm_read_time_us":333,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2824,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":215,"thread_start_us":14117,"threads_started":1,"wal-append.queue_time_us":12494}
I20260808 01:50:43.986846 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe): perf score=1.008223
I20260808 01:50:44.035135 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.212s user 0.132s sys 0.020s Metrics: {"bytes_written":13713,"cfile_init":1,"dirs.queue_time_us":100,"dirs.run_cpu_time_us":282,"dirs.run_wall_time_us":1451,"drs_written":1,"lbm_read_time_us":280,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":346,"thread_start_us":26622,"threads_started":1,"wal-append.queue_time_us":27005}
I20260808 01:50:44.042564 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 9643 bytes on disk
I20260808 01:50:44.047050 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":366,"lbm_reads_lt_1ms":4}
I20260808 01:50:44.053004 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 3487 bytes on disk
I20260808 01:50:44.055122 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":317,"lbm_reads_lt_1ms":4}
I20260808 01:50:44.071604 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.013302
I20260808 01:50:44.168569 6340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:50:44.178288 6340 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:50:44.182569 6340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:50:44.202036 6340 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:50:44.221866 6340 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:50:44.222354 6340 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260808 01:50:44.222806 6340 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:50:44.223104 6340 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:44.299145 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.474s user 0.287s sys 0.049s Metrics: {"bytes_written":31899,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19411,"cfile_cache_miss":26,"cfile_cache_miss_bytes":56727,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":88255,"dirs.run_cpu_time_us":2913,"dirs.run_wall_time_us":17997,"drs_written":1,"lbm_read_time_us":3659,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2703,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1052,"thread_start_us":122798,"threads_started":5}
I20260808 01:50:44.310228 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.060569
I20260808 01:50:44.571686 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.584s user 0.261s sys 0.034s Metrics: {"bytes_written":31747,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19342,"cfile_cache_miss":26,"cfile_cache_miss_bytes":56506,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":26113,"dirs.run_cpu_time_us":2043,"dirs.run_wall_time_us":13756,"drs_written":1,"lbm_read_time_us":5541,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":3771,"lbm_writes_lt_1ms":23,"mutex_wait_us":63,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1048,"spinlock_wait_cycles":70400,"thread_start_us":39143,"threads_started":4}
I20260808 01:50:44.579846 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe): 10187 bytes on disk
I20260808 01:50:44.581730 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(0c44eefb43024e9290fdba8647e745fe) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
I20260808 01:50:44.587036 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.030173
I20260808 01:50:44.586225 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.275s user 0.110s sys 0.002s Metrics: {"bytes_written":13342,"cfile_init":1,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":930,"drs_written":1,"lbm_read_time_us":262,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":331,"thread_start_us":16834,"threads_started":1,"wal-append.queue_time_us":9706}
I20260808 01:50:44.594215 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 10221 bytes on disk
I20260808 01:50:44.596221 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4}
I20260808 01:50:44.602625 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e): perf score=1.013049
I20260808 01:50:44.731113 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.659s user 0.273s sys 0.022s Metrics: {"bytes_written":32403,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19725,"cfile_cache_miss":26,"cfile_cache_miss_bytes":57627,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":42614,"dirs.run_cpu_time_us":2928,"dirs.run_wall_time_us":18887,"drs_written":1,"lbm_read_time_us":4515,"lbm_reads_lt_1ms":46,"lbm_write_time_us":18115,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"mutex_wait_us":38,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1069,"spinlock_wait_cycles":55552,"thread_start_us":59333,"threads_started":5}
I20260808 01:50:44.738340 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2a97be89f618431b90b49c97b2c9f001): perf score=0.060569
I20260808 01:50:44.760390 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.172s user 0.093s sys 0.010s Metrics: {"bytes_written":11640,"cfile_init":1,"dirs.queue_time_us":16502,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":8011,"drs_written":1,"lbm_read_time_us":275,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12276,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":265,"thread_start_us":24011,"threads_started":2,"wal-append.queue_time_us":7042}
I20260808 01:50:44.767216 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 2721 bytes on disk
I20260808 01:50:44.769191 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4}
I20260808 01:50:44.774778 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=1.007117
I20260808 01:50:44.929014 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.190s user 0.107s sys 0.002s Metrics: {"bytes_written":12937,"cfile_init":1,"dirs.queue_time_us":4748,"dirs.run_cpu_time_us":308,"dirs.run_wall_time_us":958,"drs_written":1,"lbm_read_time_us":316,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3357,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":315,"thread_start_us":8788,"threads_started":1,"wal-append.queue_time_us":9643}
I20260808 01:50:44.939682 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001): perf score=1.014189
I20260808 01:50:44.999315 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.389s user 0.213s sys 0.016s Metrics: {"bytes_written":29337,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17552,"cfile_cache_miss":26,"cfile_cache_miss_bytes":51712,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":15167,"dirs.run_cpu_time_us":2452,"dirs.run_wall_time_us":7452,"drs_written":1,"lbm_read_time_us":3617,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2587,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":950,"thread_start_us":36854,"threads_started":4}
I20260808 01:50:45.006285 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.030173
I20260808 01:50:45.138139 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.362s user 0.220s sys 0.021s Metrics: {"bytes_written":30939,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18686,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54810,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":31256,"dirs.run_cpu_time_us":2642,"dirs.run_wall_time_us":15805,"drs_written":1,"lbm_read_time_us":7370,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2890,"lbm_writes_lt_1ms":23,"mutex_wait_us":380,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1012,"thread_start_us":40783,"threads_started":4}
I20260808 01:50:45.144233 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(1fff9439a0bd4286827363f69521c44f): perf score=0.014976
I20260808 01:50:45.166875 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.160s user 0.099s sys 0.010s Metrics: {"bytes_written":11908,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":666,"drs_written":1,"lbm_read_time_us":264,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":276,"thread_start_us":11497,"threads_started":1,"wal-append.queue_time_us":11873}
I20260808 01:50:45.173002 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 9836 bytes on disk
I20260808 01:50:45.175239 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":301,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.180011 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05): 9425 bytes on disk
I20260808 01:50:45.181715 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(86d58fdfa95b45f68c0648a585924d05) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":278,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.188225 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 9282 bytes on disk
I20260808 01:50:45.190079 4873 maintenance_manager.cc:643] P 01adc8a4bde24946b6bf7e18e876e977: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":281,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.198938 4984 maintenance_manager.cc:419] P 01adc8a4bde24946b6bf7e18e876e977: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=1.009254
I20260808 01:50:45.226366 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.081s user 0.052s sys 0.016s Metrics: {"bytes_written":7902,"cfile_init":1,"dirs.queue_time_us":84,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":253,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":123,"thread_start_us":11542,"threads_started":1,"wal-append.queue_time_us":11884}
I20260808 01:50:45.232905 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 9836 bytes on disk
I20260808 01:50:45.234688 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.240079 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 1320 bytes on disk
I20260808 01:50:45.242975 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":274,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.255697 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f): perf score=1.001274
I20260808 01:50:45.281116 6615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.68:35651 every 8 connection(s)
I20260808 01:50:45.281762 6340 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.68:35651
I20260808 01:50:45.285682 6340 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb
I20260808 01:50:45.287854 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 6340
I20260808 01:50:45.288763 30417 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/instance
W20260808 01:50:45.376243 4978 generic_service.cc:139] {username='slave'} at 127.0.0.1:54670 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:45.376771 4978 generic_service.cc:175] {username='slave'} at 127.0.0.1:54670 changed flags via RPC: enable_tablet_copy from 'true' to 'false'
I20260808 01:50:45.390254 6617 heartbeater.cc:344] Connected to a master server at 127.29.180.126:32885
I20260808 01:50:45.390792 6617 heartbeater.cc:461] Registering TS with master...
I20260808 01:50:45.392199 6617 heartbeater.cc:507] Master 127.29.180.126:32885 requested a full tablet report, sending...
I20260808 01:50:45.400632 4784 ts_manager.cc:194] Registered new tserver with Master: c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651)
I20260808 01:50:45.402697 4784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.68:60345
W20260808 01:50:45.418674 5116 generic_service.cc:139] {username='slave'} at 127.0.0.1:51332 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:45.419203 5116 generic_service.cc:175] {username='slave'} at 127.0.0.1:51332 changed flags via RPC: enable_tablet_copy from 'true' to 'false'
I20260808 01:50:45.466084 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.524s user 0.233s sys 0.026s Metrics: {"bytes_written":30727,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18518,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54256,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":76552,"dirs.run_cpu_time_us":2566,"dirs.run_wall_time_us":15461,"drs_written":1,"lbm_read_time_us":3647,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2522,"lbm_writes_lt_1ms":23,"mutex_wait_us":279,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1003,"spinlock_wait_cycles":26496,"thread_start_us":102226,"threads_started":5}
I20260808 01:50:45.472147 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 10395 bytes on disk
I20260808 01:50:45.474375 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4}
W20260808 01:50:45.478709 5255 generic_service.cc:139] {username='slave'} at 127.0.0.1:33916 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:45.478739 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(2d7aa70246b445149411741e595f27c2): perf score=0.060569
I20260808 01:50:45.479194 5255 generic_service.cc:175] {username='slave'} at 127.0.0.1:33916 changed flags via RPC: enable_tablet_copy from 'true' to 'false'
W20260808 01:50:45.532303 6611 generic_service.cc:139] {username='slave'} at 127.0.0.1:41366 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:45.532778 6611 generic_service.cc:175] {username='slave'} at 127.0.0.1:41366 changed flags via RPC: enable_tablet_copy from 'true' to 'false'
I20260808 01:50:45.537190 30417 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu with pid 4847
I20260808 01:50:45.625960 30417 disk_failure-itest.cc:315] Restarting server with injected errors...
I20260808 01:50:45.627672 30417 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
/tmp/dist-test-taskz8giuv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.180.65:33111
--local_ip_for_outbound_sockets=127.29.180.65
--tserver_master_addrs=127.29.180.126:32885
--webserver_port=44827
--webserver_interface=127.29.180.65
--builtin_ntp_servers=127.29.180.84:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_secs=1
--flush_threshold_mb=1
--cfile_inject_corruption=1.0 with env {}
I20260808 01:50:45.654088 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.396s user 0.258s sys 0.000s Metrics: {"bytes_written":33614,"cfile_cache_hit":10,"cfile_cache_hit_bytes":20634,"cfile_cache_miss":26,"cfile_cache_miss_bytes":60134,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":6243,"dirs.run_cpu_time_us":2627,"dirs.run_wall_time_us":10764,"drs_written":1,"lbm_read_time_us":4524,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":2569,"lbm_writes_lt_1ms":23,"mutex_wait_us":71,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1119,"spinlock_wait_cycles":2432,"thread_start_us":11737,"threads_started":4}
I20260808 01:50:45.656512 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.177s user 0.142s sys 0.000s Metrics: {"bytes_written":16892,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":339,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":265,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":469,"thread_start_us":828,"threads_started":1,"wal-append.queue_time_us":1193}
I20260808 01:50:45.660651 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.003577
I20260808 01:50:45.662686 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001): 9739 bytes on disk
I20260808 01:50:45.664379 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2a97be89f618431b90b49c97b2c9f001) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":288,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.668991 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(2d7aa70246b445149411741e595f27c2): perf score=1.010025
W20260808 01:50:45.694463 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
W20260808 01:50:45.712576 5008 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:45.712996 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.052s user 0.031s sys 0.017s Metrics: {"bytes_written":5450,"cfile_init":1,"dirs.queue_time_us":654,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":804,"drs_written":1,"lbm_read_time_us":252,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2259,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":32,"thread_start_us":13379,"threads_started":1,"wal-append.queue_time_us":13772}
I20260808 01:50:45.717464 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65): perf score=0.908332
W20260808 01:50:45.787343 5008 consensus_peers.cc:597] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:45.859635 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.142s user 0.132s sys 0.008s Metrics: {"bytes_written":30293,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18281,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53287,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2382,"dirs.run_cpu_time_us":1501,"dirs.run_wall_time_us":7006,"drs_written":1,"lbm_read_time_us":2693,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2050,"lbm_writes_lt_1ms":23,"mutex_wait_us":28,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":990,"spinlock_wait_cycles":29440,"thread_start_us":1891,"threads_started":1}
I20260808 01:50:45.864090 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.002225
W20260808 01:50:45.898635 5008 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:45.903266 5147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:45.916870 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: FlushMRSOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.052s user 0.029s sys 0.016s Metrics: {"bytes_written":4924,"cfile_init":1,"dirs.queue_time_us":101,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1501,"drs_written":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1644,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":11,"thread_start_us":20812,"threads_started":1,"wal-append.queue_time_us":21176}
W20260808 01:50:45.918864 5147 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:45.917922 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.248s user 0.233s sys 0.004s Metrics: {"bytes_written":30515,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18338,"cfile_cache_miss":26,"cfile_cache_miss_bytes":53700,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":988,"dirs.run_cpu_time_us":1979,"dirs.run_wall_time_us":7061,"drs_written":1,"lbm_read_time_us":3856,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2657,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":993}
I20260808 01:50:45.922744 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f): 10789 bytes on disk
I20260808 01:50:45.923702 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2): 9670 bytes on disk
I20260808 01:50:45.924393 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(1fff9439a0bd4286827363f69521c44f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":299,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.924953 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(2d7aa70246b445149411741e595f27c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":183,"lbm_reads_lt_1ms":4}
I20260808 01:50:45.928707 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f): perf score=0.757781
I20260808 01:50:45.929366 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(9ee6f0c29217471e93394d4b9814144e): perf score=0.060569
I20260808 01:50:46.044198 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.114s user 0.086s sys 0.020s Metrics: {"bytes_written":14604,"cfile_init":1,"dirs.queue_time_us":108,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1422,"drs_written":1,"lbm_read_time_us":259,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3578,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":379,"thread_start_us":14745,"threads_started":1,"wal-append.queue_time_us":16678}
I20260808 01:50:46.050491 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e): perf score=1.013009
I20260808 01:50:46.180132 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: CompactRowSetsOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.251s user 0.244s sys 0.004s Metrics: {"bytes_written":32403,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19726,"cfile_cache_miss":26,"cfile_cache_miss_bytes":57406,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":3637,"dirs.run_cpu_time_us":2210,"dirs.run_wall_time_us":10321,"drs_written":1,"lbm_read_time_us":4063,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2999,"lbm_writes_lt_1ms":23,"mutex_wait_us":61,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1069,"spinlock_wait_cycles":23296,"thread_start_us":3516,"threads_started":3}
I20260808 01:50:46.185746 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f): 10395 bytes on disk
I20260808 01:50:46.187444 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(24ad2932d9a1489b96a63bc762f6a23f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":305,"lbm_reads_lt_1ms":4}
I20260808 01:50:46.191429 5261 maintenance_manager.cc:419] P 99d0ee4efbd948a3b143ff2297ad9fc9: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 9643 bytes on disk
I20260808 01:50:46.193037 5150 maintenance_manager.cc:643] P 99d0ee4efbd948a3b143ff2297ad9fc9: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
W20260808 01:50:46.198287 6635 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 01:50:46.198828 6635 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 01:50:46.199455 6635 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260808 01:50:46.234839 6635 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260808 01:50:46.235502 6635 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260808 01:50:46.235864 6635 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
W20260808 01:50:46.236754 6635 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.180.65
I20260808 01:50:46.240463 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.189s user 0.161s sys 0.005s Metrics: {"bytes_written":29337,"cfile_cache_hit":10,"cfile_cache_hit_bytes":17552,"cfile_cache_miss":26,"cfile_cache_miss_bytes":51712,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2893,"dirs.run_cpu_time_us":2119,"dirs.run_wall_time_us":6748,"drs_written":1,"lbm_read_time_us":3194,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2054,"lbm_writes_lt_1ms":23,"mutex_wait_us":233,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":950,"spinlock_wait_cycles":23424,"thread_start_us":2674,"threads_started":3}
I20260808 01:50:46.246516 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e): 9282 bytes on disk
I20260808 01:50:46.248248 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(9ee6f0c29217471e93394d4b9814144e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":289,"lbm_reads_lt_1ms":4}
I20260808 01:50:46.252830 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(5843450fd1f54792899cc7190c94821f): perf score=0.030173
I20260808 01:50:46.307168 6635 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--cfile_inject_corruption=1
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.180.84:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2
--fs_wal_dir=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.180.65:33111
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
--webserver_interface=127.29.180.65
--webserver_port=44827
--flush_threshold_mb=1
--flush_threshold_secs=1
--tserver_master_addrs=127.29.180.126:32885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.180.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision cb747ff0ca0209e3fa98be4167c67edd474acd67
build type FASTDEBUG
built by None at 08 Aug 2026 01:43:22 UTC on e7f111948823
build id 13235
TSAN enabled
I20260808 01:50:46.308777 6635 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:50:46.310637 6635 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:50:46.325264 6661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:50:46.353279 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.100s user 0.084s sys 0.008s Metrics: {"bytes_written":9832,"cfile_init":1,"dirs.queue_time_us":120,"dirs.run_cpu_time_us":319,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":252,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2674,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":197,"thread_start_us":13760,"threads_started":1,"wal-append.queue_time_us":14173}
I20260808 01:50:46.362336 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 2051 bytes on disk
I20260808 01:50:46.364337 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":257,"lbm_reads_lt_1ms":4}
I20260808 01:50:46.371356 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(5843450fd1f54792899cc7190c94821f): perf score=1.004074
I20260808 01:50:46.407956 6617 heartbeater.cc:499] Master 127.29.180.126:32885 was elected leader, sending a full tablet report...
I20260808 01:50:46.531423 6651 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.532529 6651 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.544332 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 203 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.549175 6610 raft_consensus.cc:493] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.549937 6610 raft_consensus.cc:515] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.553028 6610 leader_election.cc:290] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:50:46.555932 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.582262 6651 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:46.583344 5008 leader_election.cc:336] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.584156 5008 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
W20260808 01:50:46.585697 5008 leader_election.cc:336] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.586475 6651 raft_consensus.cc:2749] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:46.597692 5008 leader_election.cc:304] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:50:46.599391 6651 raft_consensus.cc:2749] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:46.687500 6651 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.688285 6651 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:46.699591 5008 leader_election.cc:336] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.702374 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.704362 5007 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:50:46.705770 6610 raft_consensus.cc:2749] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:46.706990 6651 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:50:46.748275 6651 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.749071 6651 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.758832 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.762104 6651 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:46.767446 5008 leader_election.cc:336] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.768208 5008 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
I20260808 01:50:46.770040 6651 raft_consensus.cc:2749] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:46.777112 6659 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.778002 6659 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.782415 6659 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:46.782737 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 203 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:46.783576 5074 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:50:46.785087 5146 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:46.785951 6659 raft_consensus.cc:2804] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:46.786319 6659 raft_consensus.cc:493] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.786649 6659 raft_consensus.cc:3060] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:46.787120 5147 leader_election.cc:336] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.797000 6658 raft_consensus.cc:493] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.797677 6658 raft_consensus.cc:515] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.798465 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.426s user 0.336s sys 0.034s Metrics: {"bytes_written":30939,"cfile_cache_hit":10,"cfile_cache_hit_bytes":18685,"cfile_cache_miss":26,"cfile_cache_miss_bytes":54651,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":17623,"dirs.run_cpu_time_us":2464,"dirs.run_wall_time_us":11982,"drs_written":1,"lbm_read_time_us":4640,"lbm_reads_lt_1ms":46,"lbm_write_time_us":3476,"lbm_writes_lt_1ms":23,"mutex_wait_us":8347,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1012,"thread_start_us":52739,"threads_started":4}
I20260808 01:50:46.801004 6658 leader_election.cc:290] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:46.800976 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:46.803613 6659 raft_consensus.cc:515] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.808580 6659 leader_election.cc:290] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:46.808759 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d7aa70246b445149411741e595f27c2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 203 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:46.809276 5074 raft_consensus.cc:3060] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:46.815950 5147 leader_election.cc:336] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.816592 5147 leader_election.cc:304] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9; no voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356
I20260808 01:50:46.819708 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2): perf score=0.014976
I20260808 01:50:46.820801 6658 raft_consensus.cc:2749] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:46.823972 6651 raft_consensus.cc:493] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.824730 6651 raft_consensus.cc:515] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.827399 6658 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.827946 6658 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.830210 6658 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:46.832036 5074 raft_consensus.cc:2468] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 2.
W20260808 01:50:46.835124 5008 leader_election.cc:336] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.841169 6651 leader_election.cc:290] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:46.846165 5147 leader_election.cc:336] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.851902 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.852809 5212 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:50:46.864450 5007 leader_election.cc:304] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:46.865640 6651 raft_consensus.cc:2804] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:46.866103 6651 raft_consensus.cc:493] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.866562 6651 raft_consensus.cc:3060] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:46.867064 5147 leader_election.cc:336] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.875926 6651 raft_consensus.cc:515] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.880777 6651 leader_election.cc:290] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:46.883862 5008 leader_election.cc:336] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
W20260808 01:50:46.886405 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 78295us
I20260808 01:50:46.888470 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:46.889024 5074 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:50:46.890192 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86d58fdfa95b45f68c0648a585924d05" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:50:46.890995 5212 raft_consensus.cc:3060] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:46.890897 5146 leader_election.cc:304] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:46.892114 6659 raft_consensus.cc:2804] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:46.892741 6659 raft_consensus.cc:697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:50:46.893751 6659 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 1.203, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.895963 6658 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.896453 6658 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.899597 6658 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:46.900589 6610 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.901346 6610 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.903944 5146 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:46.905879 6658 raft_consensus.cc:2804] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:46.906970 6610 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:50:46.906263 6658 raft_consensus.cc:493] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.911643 6658 raft_consensus.cc:3060] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:46.912277 5008 leader_election.cc:336] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.915131 5211 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" is_pre_election: true
I20260808 01:50:46.916045 5211 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 1.
I20260808 01:50:46.921849 5007 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:46.923076 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:46.923640 6651 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:46.924620 5074 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:50:46.926230 5146 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:46.926306 6651 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.927156 6651 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:46.927829 6671 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:46.928392 6671 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:46.928896 6671 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:46.937598 5147 leader_election.cc:336] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.940726 5212 raft_consensus.cc:2468] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65d57b7135430eb6695032355bd356 in term 2.
I20260808 01:50:46.943604 6658 raft_consensus.cc:515] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.946012 5007 leader_election.cc:304] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:46.947695 6610 raft_consensus.cc:2804] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:46.955063 6610 raft_consensus.cc:697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Becoming Leader. State: Replica: ec65d57b7135430eb6695032355bd356, State: Running, Role: LEADER
I20260808 01:50:46.956149 6610 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.958465 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 1 to 2, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:46.968935 6651 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:46.976357 6651 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:50:46.972692 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: FlushMRSOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.152s user 0.063s sys 0.015s Metrics: {"bytes_written":7489,"cfile_init":1,"dirs.queue_time_us":8867,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":1552,"drs_written":1,"lbm_read_time_us":319,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2645,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":107,"thread_start_us":29386,"threads_started":1,"wal-append.queue_time_us":29807}
W20260808 01:50:46.989372 5008 leader_election.cc:336] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:46.980258 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: term changed from 1 to 2, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to ec65d57b7135430eb6695032355bd356 (127.29.180.66). New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:46.993989 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2): perf score=1.010230
I20260808 01:50:46.975443 5212 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "ec65d57b7135430eb6695032355bd356" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9"
I20260808 01:50:46.997678 6659 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:47.013777 6675 raft_consensus.cc:493] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:47.011529 6659 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.014979 6671 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.016234 5212 raft_consensus.cc:2393] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ec65d57b7135430eb6695032355bd356 in current term 2: Already voted for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in this term.
I20260808 01:50:47.023459 6671 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:47.023190 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:47.024116 5074 raft_consensus.cc:2393] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in current term 2: Already voted for candidate ec65d57b7135430eb6695032355bd356 in this term.
I20260808 01:50:47.027169 5007 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977, 99d0ee4efbd948a3b143ff2297ad9fc9
W20260808 01:50:47.036139 5147 leader_election.cc:336] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.036995 5147 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9; no voters: 01adc8a4bde24946b6bf7e18e876e977, ec65d57b7135430eb6695032355bd356
I20260808 01:50:47.038152 6651 raft_consensus.cc:2749] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:47.039489 6671 raft_consensus.cc:2749] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:47.040835 6658 leader_election.cc:290] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:47.042467 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ad2932d9a1489b96a63bc762f6a23f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:47.043594 5074 raft_consensus.cc:3060] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:47.053226 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:47.054219 5073 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:50:47.056403 5146 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:47.057786 6658 raft_consensus.cc:2804] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:47.058297 6658 raft_consensus.cc:493] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:47.058771 6658 raft_consensus.cc:3060] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:47.013839 6672 log.cc:927] Time spent T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260808 01:50:47.066653 6659 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:47.076581 5147 leader_election.cc:336] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.014729 6675 raft_consensus.cc:515] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.088177 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:47.089119 5073 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 1.
I20260808 01:50:47.091157 5146 leader_election.cc:304] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:47.092597 6659 raft_consensus.cc:2804] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:47.093221 6659 raft_consensus.cc:493] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 01adc8a4bde24946b6bf7e18e876e977)
I20260808 01:50:47.096695 6659 raft_consensus.cc:3060] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:47.107972 5147 leader_election.cc:336] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.113099 5074 raft_consensus.cc:2468] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 2.
I20260808 01:50:47.108783 6675 leader_election.cc:290] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:47.120294 5147 leader_election.cc:336] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.130127 5146 leader_election.cc:304] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:47.132892 6675 raft_consensus.cc:2804] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:47.134706 6675 raft_consensus.cc:697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:50:47.136598 6675 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.116711 6658 raft_consensus.cc:515] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.145753 5074 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d367581a54c433ea4f227bd4f479ed2" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:47.146718 5074 raft_consensus.cc:3060] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:47.156628 6658 leader_election.cc:290] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:47.177755 5147 leader_election.cc:336] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.193955 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 1 to 2, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:47.185556 6659 raft_consensus.cc:515] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.203464 5074 raft_consensus.cc:2468] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 2.
W20260808 01:50:47.219622 5147 leader_election.cc:336] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111)
I20260808 01:50:47.220363 6659 leader_election.cc:290] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:47.218895 5073 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c44eefb43024e9290fdba8647e745fe" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:47.224220 5073 raft_consensus.cc:3060] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:47.237521 5146 leader_election.cc:304] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:47.239494 6659 raft_consensus.cc:2804] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:47.241438 6659 raft_consensus.cc:697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:50:47.243089 6659 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.246393 5073 raft_consensus.cc:2468] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 2.
I20260808 01:50:47.249800 5146 leader_election.cc:304] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters: 01adc8a4bde24946b6bf7e18e876e977
I20260808 01:50:47.251787 6675 raft_consensus.cc:2804] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:47.252490 6675 raft_consensus.cc:697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:50:47.253723 6675 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:47.301067 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 1 to 2, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:47.345193 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 1 to 2, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:47.393280 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20260808 01:50:47.396068 6659 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.001s
I20260808 01:50:47.436215 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 2 index: 205. (index mismatch)
I20260808 01:50:47.437490 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: CompactRowSetsOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.442s user 0.293s sys 0.034s Metrics: {"bytes_written":31899,"cfile_cache_hit":10,"cfile_cache_hit_bytes":19415,"cfile_cache_miss":26,"cfile_cache_miss_bytes":56699,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":28709,"dirs.run_cpu_time_us":2713,"dirs.run_wall_time_us":15891,"drs_written":1,"lbm_read_time_us":22585,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":3762,"lbm_writes_lt_1ms":23,"mutex_wait_us":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":1052,"thread_start_us":51874,"threads_started":4}
I20260808 01:50:47.448846 6610 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.001s
W20260808 01:50:47.454033 5147 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:47.483855 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2): 10221 bytes on disk
I20260808 01:50:47.486464 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(8d367581a54c433ea4f227bd4f479ed2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":300,"lbm_reads_lt_1ms":4}
I20260808 01:50:47.493777 5122 maintenance_manager.cc:419] P ec65d57b7135430eb6695032355bd356: Scheduling UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f): 9836 bytes on disk
I20260808 01:50:47.516036 5011 maintenance_manager.cc:643] P ec65d57b7135430eb6695032355bd356: UndoDeltaBlockGCOp(5843450fd1f54792899cc7190c94821f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":303,"lbm_reads_lt_1ms":4}
W20260808 01:50:47.535813 5008 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:47.549214 5147 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:47.628671 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20260808 01:50:47.630236 6658 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20260808 01:50:47.663132 5147 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:47.753118 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20260808 01:50:47.755568 6658 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
W20260808 01:50:47.779992 5147 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:47.791811 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20260808 01:50:47.793181 6658 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20260808 01:50:47.729794 6660 debug-util.cc:398] Leaking SignalData structure 0x7b08000375c0 after lost signal to thread 6635
W20260808 01:50:47.868332 6635 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.542s user 0.439s sys 0.730s
W20260808 01:50:46.327884 6662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:50:47.868742 6635 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.543s user 0.439s sys 0.730s
W20260808 01:50:47.879878 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:50:47.879989 6635 server_base.cc:1061] running on GCE node
I20260808 01:50:47.882012 6635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 01:50:47.884196 6635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 01:50:47.885569 6635 hybrid_clock.cc:648] HybridClock initialized: now 1786153847885521 us; error 74 us; skew 500 ppm
I20260808 01:50:47.886502 6635 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:50:47.893124 6635 webserver.cc:533] Webserver started at http://127.29.180.65:44827/ using document root <none> and password file <none>
I20260808 01:50:47.895011 6635 fs_manager.cc:362] Metadata directory not provided
I20260808 01:50:47.895292 6635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:50:47.911454 6635 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.006s sys 0.006s
W20260808 01:50:47.914920 5008 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:47.968641 6712 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:47.969578 6712 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:47.970907 6713 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 01:50:47.972692 6635 fs_manager.cc:730] Time spent opening block manager: real 0.057s user 0.005s sys 0.002s
I20260808 01:50:47.973006 6635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2,/tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
uuid: "01adc8a4bde24946b6bf7e18e876e977"
format_stamp: "Formatted at 2026-08-08 01:49:49 on dist-test-slave-p3t5"
I20260808 01:50:47.975715 6635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/wal
3 data directories: /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-2/data
Total live blocks: 55
Total live bytes: 408273
Total live bytes (after alignment): 528384
Total number of LBM containers: 17 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:50:48.058669 6635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260808 01:50:48.060768 6635 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:50:48.061335 6635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:50:48.065040 6635 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:50:48.073467 6720 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260808 01:50:48.082906 5147 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:48.117307 6635 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260808 01:50:48.117657 6635 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.046s user 0.009s sys 0.015s
I20260808 01:50:48.117995 6635 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260808 01:50:48.124298 6721 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:50:48.130792 6720 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:50:48.138335 6722 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
W20260808 01:50:48.139256 6721 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260808 01:50:48.139530 6721 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260808 01:50:48.139830 6721 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442784
E20260808 01:50:48.140497 6721 tablet.cc:439] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 12100708394399442784: checksum does not match: 3786038964 vs expected 3786038964
E20260808 01:50:48.140702 6720 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442700
E20260808 01:50:48.141433 6720 tablet.cc:439] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 12100708394399442700: checksum does not match: 652593189 vs expected 652593189
E20260808 01:50:48.141916 6721 ts_tablet_manager.cc:1425] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442784: checksum does not match: 3786038964 vs expected 3786038964
I20260808 01:50:48.142220 6721 ts_tablet_manager.cc:1403] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20260808 01:50:48.142514 6721 tablet_replica.cc:333] stopping tablet replica
E20260808 01:50:48.142756 6720 ts_tablet_manager.cc:1425] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442700: checksum does not match: 652593189 vs expected 652593189
I20260808 01:50:48.143260 6720 ts_tablet_manager.cc:1403] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260808 01:50:48.143576 6720 tablet_replica.cc:333] stopping tablet replica
E20260808 01:50:48.144867 6722 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442760
E20260808 01:50:48.145354 6722 tablet.cc:439] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 12100708394399442760: checksum does not match: 837457110 vs expected 837457110
E20260808 01:50:48.146241 6722 ts_tablet_manager.cc:1425] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442760: checksum does not match: 837457110 vs expected 837457110
I20260808 01:50:48.142835 6721 raft_consensus.cc:2243] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.143862 6720 raft_consensus.cc:2243] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.146871 6721 raft_consensus.cc:2272] T 8d367581a54c433ea4f227bd4f479ed2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.147071 6720 raft_consensus.cc:2272] T 2a97be89f618431b90b49c97b2c9f001 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.146526 6722 ts_tablet_manager.cc:1403] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260808 01:50:48.148749 6722 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.149014 6722 raft_consensus.cc:2243] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.149336 6722 raft_consensus.cc:2272] T 5843450fd1f54792899cc7190c94821f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.151124 6722 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
I20260808 01:50:48.157193 6720 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.157467 6722 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442748
E20260808 01:50:48.157982 6722 tablet.cc:439] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(6): Corruption: failed to init CFileReader for block 12100708394399442748: checksum does not match: 789294655 vs expected 789294655
E20260808 01:50:48.158887 6722 ts_tablet_manager.cc:1425] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442748: checksum does not match: 789294655 vs expected 789294655
I20260808 01:50:48.159168 6722 ts_tablet_manager.cc:1403] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 01:50:48.159441 6722 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.159686 6722 raft_consensus.cc:2243] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.159996 6722 raft_consensus.cc:2272] T 213cfcc36a754065ab9203bf5a8f6f65 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
E20260808 01:50:48.163663 6720 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442796
E20260808 01:50:48.164153 6720 tablet.cc:439] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 12100708394399442796: checksum does not match: 1522150044 vs expected 1522150044
I20260808 01:50:48.164758 6722 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.165675 6720 ts_tablet_manager.cc:1425] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442796: checksum does not match: 1522150044 vs expected 1522150044
I20260808 01:50:48.165952 6720 ts_tablet_manager.cc:1403] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 01:50:48.167101 6720 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.167351 6720 raft_consensus.cc:2243] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.167666 6720 raft_consensus.cc:2272] T 9ee6f0c29217471e93394d4b9814144e P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
E20260808 01:50:48.171101 6722 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442688
E20260808 01:50:48.171634 6722 tablet.cc:439] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 12100708394399442688: checksum does not match: 1144801091 vs expected 1144801091
I20260808 01:50:48.172338 6720 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.172494 6722 ts_tablet_manager.cc:1425] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442688: checksum does not match: 1144801091 vs expected 1144801091
I20260808 01:50:48.172791 6722 ts_tablet_manager.cc:1403] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 01:50:48.173046 6722 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.173272 6722 raft_consensus.cc:2243] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.173593 6722 raft_consensus.cc:2272] T 24ad2932d9a1489b96a63bc762f6a23f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.176669 6722 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.178707 6720 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442712
E20260808 01:50:48.179282 6720 tablet.cc:439] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(6): Corruption: failed to init CFileReader for block 12100708394399442712: checksum does not match: 1814835300 vs expected 1814835300
E20260808 01:50:48.180256 6720 ts_tablet_manager.cc:1425] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442712: checksum does not match: 1814835300 vs expected 1814835300
I20260808 01:50:48.180651 6720 ts_tablet_manager.cc:1403] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 01:50:48.180963 6720 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.181242 6720 raft_consensus.cc:2243] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.181655 6720 raft_consensus.cc:2272] T 0c44eefb43024e9290fdba8647e745fe P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.182614 6721 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.182837 6722 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442724
E20260808 01:50:48.183315 6722 tablet.cc:439] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(6): Corruption: failed to init CFileReader for block 12100708394399442724: checksum does not match: 2954062558 vs expected 2954062558
E20260808 01:50:48.184188 6722 ts_tablet_manager.cc:1425] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442724: checksum does not match: 2954062558 vs expected 2954062558
I20260808 01:50:48.184475 6722 ts_tablet_manager.cc:1403] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260808 01:50:48.184721 6722 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.185118 6722 raft_consensus.cc:2243] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.185441 6722 raft_consensus.cc:2272] T 1fff9439a0bd4286827363f69521c44f P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:50:48.187495 6635 ts_tablet_manager.cc:616] Registered 10 tablets
I20260808 01:50:48.187762 6635 ts_tablet_manager.cc:595] Time spent register tablets: real 0.070s user 0.053s sys 0.000s
I20260808 01:50:48.187810 6720 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Bootstrap starting.
E20260808 01:50:48.188931 6721 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442736
E20260808 01:50:48.189515 6721 tablet.cc:439] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 12100708394399442736: checksum does not match: 2102021301 vs expected 2102021301
E20260808 01:50:48.190474 6721 ts_tablet_manager.cc:1425] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442736: checksum does not match: 2102021301 vs expected 2102021301
I20260808 01:50:48.190832 6721 ts_tablet_manager.cc:1403] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260808 01:50:48.191133 6721 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.191416 6721 raft_consensus.cc:2243] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.191818 6721 raft_consensus.cc:2272] T 2d7aa70246b445149411741e595f27c2 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
E20260808 01:50:48.195139 6720 cfile_reader.cc:647] Encountered corrupted CFile in filesystem block: 12100708394399442772
I20260808 01:50:48.195420 6720 ts_tablet_manager.cc:1993] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: failing tablet
E20260808 01:50:48.196331 6720 tablet.cc:439] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 12100708394399442772: checksum does not match: 451298314 vs expected 451298314
E20260808 01:50:48.197269 6720 ts_tablet_manager.cc:1425] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 12100708394399442772: checksum does not match: 451298314 vs expected 451298314
I20260808 01:50:48.197678 6720 ts_tablet_manager.cc:1403] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20260808 01:50:48.197961 6720 tablet_replica.cc:333] stopping tablet replica
I20260808 01:50:48.198205 6720 raft_consensus.cc:2243] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:50:48.198570 6720 raft_consensus.cc:2272] T 86d58fdfa95b45f68c0648a585924d05 P 01adc8a4bde24946b6bf7e18e876e977 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 01:50:48.321784 5008 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Status: Network error: Client connection negotiation failed: client connection to 127.29.180.65:33111: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:48.366132 6635 rpc_server.cc:307] RPC server started. Bound to: 127.29.180.65:33111
I20260808 01:50:48.366827 6831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.180.65:33111 every 8 connection(s)
I20260808 01:50:48.390434 6635 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb
I20260808 01:50:48.391433 30417 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskz8giuv/build/tsan/bin/kudu as pid 6635
I20260808 01:50:48.434727 6832 heartbeater.cc:344] Connected to a master server at 127.29.180.126:32885
I20260808 01:50:48.435316 6832 heartbeater.cc:461] Registering TS with master...
I20260808 01:50:48.436841 6832 heartbeater.cc:507] Master 127.29.180.126:32885 requested a full tablet report, sending...
I20260808 01:50:48.441722 4784 ts_manager.cc:194] Re-registered known tserver with Master: 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111)
W20260808 01:50:48.446642 4784 catalog_manager.cc:5610] Tablet 2a97be89f618431b90b49c97b2c9f001 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442700: checksum does not match: 652593189 vs expected 652593189
W20260808 01:50:48.447192 4784 catalog_manager.cc:5610] Tablet 2d7aa70246b445149411741e595f27c2 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442736: checksum does not match: 2102021301 vs expected 2102021301
W20260808 01:50:48.447542 4784 catalog_manager.cc:5610] Tablet 5843450fd1f54792899cc7190c94821f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442760: checksum does not match: 837457110 vs expected 837457110
W20260808 01:50:48.447886 4784 catalog_manager.cc:5610] Tablet 24ad2932d9a1489b96a63bc762f6a23f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442688: checksum does not match: 1144801091 vs expected 1144801091
W20260808 01:50:48.448220 4784 catalog_manager.cc:5610] Tablet 213cfcc36a754065ab9203bf5a8f6f65 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442748: checksum does not match: 789294655 vs expected 789294655
W20260808 01:50:48.448546 4784 catalog_manager.cc:5610] Tablet 1fff9439a0bd4286827363f69521c44f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442724: checksum does not match: 2954062558 vs expected 2954062558
W20260808 01:50:48.448877 4784 catalog_manager.cc:5610] Tablet 9ee6f0c29217471e93394d4b9814144e (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442796: checksum does not match: 1522150044 vs expected 1522150044
W20260808 01:50:48.449213 4784 catalog_manager.cc:5610] Tablet 8d367581a54c433ea4f227bd4f479ed2 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442784: checksum does not match: 3786038964 vs expected 3786038964
W20260808 01:50:48.449538 4784 catalog_manager.cc:5610] Tablet 0c44eefb43024e9290fdba8647e745fe (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442712: checksum does not match: 1814835300 vs expected 1814835300
W20260808 01:50:48.449908 4784 catalog_manager.cc:5610] Tablet 86d58fdfa95b45f68c0648a585924d05 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442772: checksum does not match: 451298314 vs expected 451298314
I20260808 01:50:48.455320 4784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.180.65:34227
I20260808 01:50:48.474128 30417 disk_failure-itest.cc:298] Currently has 10 failed tablets
I20260808 01:50:48.474395 30417 disk_failure-itest.cc:274] Resetting error injection flags
W20260808 01:50:48.489476 6827 generic_service.cc:139] {username='slave'} at 127.0.0.1:54906 forcing change of non-runtime-safe flag env_inject_eio
I20260808 01:50:48.489953 6827 generic_service.cc:175] {username='slave'} at 127.0.0.1:54906 changed flags via RPC: env_inject_eio from '0' to '0.0'
W20260808 01:50:48.492851 6827 generic_service.cc:139] {username='slave'} at 127.0.0.1:54906 forcing change of non-runtime-safe flag cfile_inject_corruption
I20260808 01:50:48.493139 6827 generic_service.cc:175] {username='slave'} at 127.0.0.1:54906 changed flags via RPC: cfile_inject_corruption from '1' to '0.0'
W20260808 01:50:48.495597 6827 generic_service.cc:139] {username='slave'} at 127.0.0.1:54906 forcing change of non-runtime-safe flag tablet_inject_kudu_2233
I20260808 01:50:48.495920 6827 generic_service.cc:175] {username='slave'} at 127.0.0.1:54906 changed flags via RPC: tablet_inject_kudu_2233 from '0' to '0.0'
W20260808 01:50:48.498209 6827 generic_service.cc:139] {username='slave'} at 127.0.0.1:54906 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:48.498522 6827 generic_service.cc:175] {username='slave'} at 127.0.0.1:54906 changed flags via RPC: enable_tablet_copy from 'true' to 'true'
W20260808 01:50:48.501127 5116 generic_service.cc:139] {username='slave'} at 127.0.0.1:51332 forcing change of non-runtime-safe flag env_inject_eio
I20260808 01:50:48.530737 5008 consensus_queue.cc:579] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
W20260808 01:50:48.531284 5008 consensus_peers.cc:597] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: failed to init CFileReader for block 12100708394399442748: checksum does not match: 789294655 vs expected 789294655. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:48.538606 5147 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.542493 5073 consensus_queue.cc:237] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.548245 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 2.204, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.554699 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 204. Preceding OpId from leader: term: 2 index: 205. (index mismatch)
I20260808 01:50:48.556116 6658 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
I20260808 01:50:48.598824 5147 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.610376 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.614486 5147 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.618664 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20260808 01:50:48.620528 6839 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20260808 01:50:48.626521 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.632472 6658 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:48.639467 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:48.665267 5147 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.686095 5008 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.688261 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 1 index: 205. (index mismatch)
I20260808 01:50:48.690708 6843 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.001s
I20260808 01:50:48.708732 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.710762 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:48.720355 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:48.728458 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20260808 01:50:48.731858 6845 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:48.740365 6658 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:48.752228 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:48.772198 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:48.777158 6843 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.777891 6843 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.781781 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 3 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356" is_pre_election: true
I20260808 01:50:48.782373 5072 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 2.
I20260808 01:50:48.783543 5146 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:48.784422 6839 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:48.784746 6839 raft_consensus.cc:493] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.785040 6839 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:48.786005 6843 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:48.793567 5147 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.795146 6839 raft_consensus.cc:515] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.798729 5072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 3 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ec65d57b7135430eb6695032355bd356"
I20260808 01:50:48.799257 5072 raft_consensus.cc:3060] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:48.806824 5072 raft_consensus.cc:2468] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99d0ee4efbd948a3b143ff2297ad9fc9 in term 3.
I20260808 01:50:48.807772 6839 leader_election.cc:290] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 election: Requested vote from peers 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111), ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:48.810424 5146 leader_election.cc:304] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99d0ee4efbd948a3b143ff2297ad9fc9, ec65d57b7135430eb6695032355bd356; no voters:
I20260808 01:50:48.811695 6839 raft_consensus.cc:2804] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:48.799002 6785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 3 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977"
I20260808 01:50:48.812932 6839 raft_consensus.cc:697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Becoming Leader. State: Replica: 99d0ee4efbd948a3b143ff2297ad9fc9, State: Running, Role: LEADER
I20260808 01:50:48.813771 6839 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:48.815114 5147 leader_election.cc:343] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Illegal state: cannot vote while shut down
I20260808 01:50:48.783352 6786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fff9439a0bd4286827363f69521c44f" candidate_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" candidate_term: 3 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "01adc8a4bde24946b6bf7e18e876e977" is_pre_election: true
W20260808 01:50:48.818557 5147 leader_election.cc:343] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Illegal state: cannot vote while shut down
I20260808 01:50:48.845103 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:48.856913 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:48.859483 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: term changed from 1 to 3, leader changed from 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) to 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67). New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01adc8a4bde24946b6bf7e18e876e977" member_type: VOTER last_known_addr { host: "127.29.180.65" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:48.864195 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20260808 01:50:48.866155 6658 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:48.874197 6658 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:48.879316 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:48.892742 5008 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.895637 5008 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
I20260808 01:50:48.904026 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:48.906435 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:50:48.910991 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24ad2932d9a1489b96a63bc762f6a23f. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:48.913582 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c44eefb43024e9290fdba8647e745fe. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:48.913884 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20260808 01:50:48.916142 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:48.916893 6839 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:48.932225 6845 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:48.934731 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:48.936031 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:48.949967 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:48.954401 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20260808 01:50:48.961174 6845 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20260808 01:50:48.970794 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:49.248042 5147 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer 01adc8a4bde24946b6bf7e18e876e977 has failed
W20260808 01:50:49.248595 5147 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Couldn't send request to peer 01adc8a4bde24946b6bf7e18e876e977. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: failed to init CFileReader for block 12100708394399442724: checksum does not match: 2954062558 vs expected 2954062558. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:49.343259 6658 consensus_peers.cc:313] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:49.384025 6658 consensus_peers.cc:313] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
I20260808 01:50:49.460225 6832 heartbeater.cc:499] Master 127.29.180.126:32885 was elected leader, sending a full tablet report...
W20260808 01:50:49.469583 4782 catalog_manager.cc:5610] Tablet 2a97be89f618431b90b49c97b2c9f001 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442700: checksum does not match: 652593189 vs expected 652593189
W20260808 01:50:49.470564 4782 catalog_manager.cc:5610] Tablet 2d7aa70246b445149411741e595f27c2 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442736: checksum does not match: 2102021301 vs expected 2102021301
W20260808 01:50:49.471160 4782 catalog_manager.cc:5610] Tablet 5843450fd1f54792899cc7190c94821f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442760: checksum does not match: 837457110 vs expected 837457110
W20260808 01:50:49.471659 4782 catalog_manager.cc:5610] Tablet 24ad2932d9a1489b96a63bc762f6a23f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442688: checksum does not match: 1144801091 vs expected 1144801091
W20260808 01:50:49.472180 4782 catalog_manager.cc:5610] Tablet 213cfcc36a754065ab9203bf5a8f6f65 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442748: checksum does not match: 789294655 vs expected 789294655
W20260808 01:50:49.472685 4782 catalog_manager.cc:5610] Tablet 1fff9439a0bd4286827363f69521c44f (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442724: checksum does not match: 2954062558 vs expected 2954062558
W20260808 01:50:49.473198 4782 catalog_manager.cc:5610] Tablet 9ee6f0c29217471e93394d4b9814144e (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442796: checksum does not match: 1522150044 vs expected 1522150044
W20260808 01:50:49.473711 4782 catalog_manager.cc:5610] Tablet 8d367581a54c433ea4f227bd4f479ed2 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442784: checksum does not match: 3786038964 vs expected 3786038964
W20260808 01:50:49.474231 4782 catalog_manager.cc:5610] Tablet 0c44eefb43024e9290fdba8647e745fe (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442712: checksum does not match: 1814835300 vs expected 1814835300
W20260808 01:50:49.474684 4782 catalog_manager.cc:5610] Tablet 86d58fdfa95b45f68c0648a585924d05 (table test-workload [id=dba8af4345154a86bdfc051c4948fd47]) has failed on TS 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65:33111): Corruption: failed to init CFileReader for block 12100708394399442772: checksum does not match: 451298314 vs expected 451298314
I20260808 01:50:49.508278 5116 generic_service.cc:175] {username='slave'} at 127.0.0.1:51332 changed flags via RPC: env_inject_eio from '0' to '0.0'
I20260808 01:50:49.509002 5116 rpcz_store.cc:275] Call kudu.server.GenericService.SetFlag from 127.0.0.1:51332 (request call id 1) took 1008 ms. Trace:
W20260808 01:50:49.592422 5075 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.008s sys 0.004s
W20260808 01:50:49.595270 5075 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.008s sys 0.004s
W20260808 01:50:49.590898 5071 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.039s sys 0.038s
W20260808 01:50:49.602183 5071 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.040s sys 0.038s
W20260808 01:50:49.594689 5074 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.002s sys 0.009s
W20260808 01:50:49.603366 5074 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.042s user 0.002s sys 0.009s
W20260808 01:50:49.601999 5069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s user 0.006s sys 0.009s
W20260808 01:50:49.603955 5069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.907s user 0.007s sys 0.009s
W20260808 01:50:49.594244 5072 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.000s sys 0.011s
W20260808 01:50:49.607314 5072 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s user 0.000s sys 0.011s
W20260808 01:50:49.606916 5008 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.005s sys 0.009s
W20260808 01:50:49.608404 5008 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.005s sys 0.009s
W20260808 01:50:49.608592 6693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.000s sys 0.049s
W20260808 01:50:49.603399 6610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.014s sys 0.031s
I20260808 01:50:49.591236 5116 rpcz_store.cc:276] 0808 01:50:48.500580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.500800 (+ 220us) service_pool.cc:224] Handling call
0808 01:50:49.508902 (+1008102us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:49.609076 6693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.000s sys 0.049s
W20260808 01:50:49.615038 5010 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.000s sys 0.011s
W20260808 01:50:49.623167 5010 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.000s sys 0.011s
I20260808 01:50:49.624375 5074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.29.180.67:38037 (request call id 412) took 1066 ms. Trace:
I20260808 01:50:49.624786 5074 rpcz_store.cc:276] 0808 01:50:48.558337 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.558533 (+ 196us) service_pool.cc:224] Handling call
0808 01:50:48.559299 (+ 766us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:48.559841 (+ 542us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:48.559901 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:48.561104 (+ 1203us) log.cc:844] Serialized 381 byte log entry
0808 01:50:49.603762 (+1042658us) raft_consensus.cc:1645] Marking committed up to 204
0808 01:50:49.603823 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:49.606256 (+ 2433us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:49.624175 (+ 17919us) raft_consensus.cc:1685] finished
0808 01:50:49.624192 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:49.624313 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1042250,"threads_started":1,"wal-append.queue_time_us":1061392}
W20260808 01:50:49.610984 6610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.015s sys 0.031s
W20260808 01:50:49.631522 5073 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.087s user 0.007s sys 0.009s
W20260808 01:50:49.631834 5073 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.087s user 0.007s sys 0.009s
W20260808 01:50:49.632452 5009 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.005s sys 0.010s
W20260808 01:50:49.632756 5009 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.005s sys 0.011s
I20260808 01:50:49.636860 6845 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.642920 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 3 index: 206. (index mismatch)
I20260808 01:50:49.646855 6862 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:50:49.671955 5116 generic_service.cc:139] {username='slave'} at 127.0.0.1:51332 forcing change of non-runtime-safe flag cfile_inject_corruption
I20260808 01:50:49.672434 5116 generic_service.cc:175] {username='slave'} at 127.0.0.1:51332 changed flags via RPC: cfile_inject_corruption from '0' to '0.0'
I20260808 01:50:49.673821 6866 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.676107 5211 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.682510 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:49.683279 6853 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.684616 6862 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20260808 01:50:49.685335 6845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35508 (request call id 12) took 1061 ms. Trace:
I20260808 01:50:49.685760 6845 rpcz_store.cc:276] 0808 01:50:48.624120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.624294 (+ 174us) service_pool.cc:224] Handling call
0808 01:50:48.628910 (+ 4616us) log.cc:844] Serialized 381 byte log entry
0808 01:50:49.685266 (+1056356us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":177,"raft.run_cpu_time_us":910,"raft.run_wall_time_us":911,"thread_start_us":36930,"threads_started":1,"wal-append.queue_time_us":30168,"wal-append.run_cpu_time_us":1795,"wal-append.run_wall_time_us":1001810}
I20260808 01:50:49.687767 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:49.686277 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.687758 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.687834 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.693276 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:49.694733 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:49.714516 6866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35508 (request call id 10) took 1168 ms. Trace:
I20260808 01:50:49.714912 6866 rpcz_store.cc:276] 0808 01:50:48.545904 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.546034 (+ 130us) service_pool.cc:224] Handling call
0808 01:50:48.550093 (+ 4059us) log.cc:844] Serialized 381 byte log entry
0808 01:50:49.714444 (+1164351us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":156,"raft.run_cpu_time_us":838,"raft.run_wall_time_us":837,"thread_start_us":1611,"threads_started":1,"wal-append.queue_time_us":1918,"wal-append.run_cpu_time_us":1237,"wal-append.run_wall_time_us":1001244}
I20260808 01:50:49.716758 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:49.721499 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 2.205, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:49.724450 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:49.727844 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 1.204, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:49.729429 5116 generic_service.cc:139] {username='slave'} at 127.0.0.1:51332 forcing change of non-runtime-safe flag tablet_inject_kudu_2233
I20260808 01:50:49.729943 6853 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49786 (request call id 10) took 1189 ms. Trace:
I20260808 01:50:49.730018 5116 generic_service.cc:175] {username='slave'} at 127.0.0.1:51332 changed flags via RPC: tablet_inject_kudu_2233 from '0' to '0.0'
I20260808 01:50:49.730356 6853 rpcz_store.cc:276] 0808 01:50:48.540449 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.540558 (+ 109us) service_pool.cc:224] Handling call
0808 01:50:48.544245 (+ 3687us) log.cc:844] Serialized 381 byte log entry
0808 01:50:49.729838 (+1185593us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1087655,"threads_started":1,"wal-append.queue_time_us":1068566}
I20260808 01:50:49.734064 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 213cfcc36a754065ab9203bf5a8f6f65 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:50:49.737273 5116 generic_service.cc:139] {username='slave'} at 127.0.0.1:51332 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:49.738343 5116 generic_service.cc:175] {username='slave'} at 127.0.0.1:51332 changed flags via RPC: enable_tablet_copy from 'false' to 'true'
W20260808 01:50:49.742003 5255 generic_service.cc:139] {username='slave'} at 127.0.0.1:33916 forcing change of non-runtime-safe flag env_inject_eio
I20260808 01:50:49.746770 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.747493 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 204. Preceding OpId from leader: term: 1 index: 205. (index mismatch)
I20260808 01:50:49.749728 6855 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.001s
I20260808 01:50:49.763346 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:49.766548 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:49.773968 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20260808 01:50:49.776150 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20260808 01:50:49.776371 6867 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:49.777663 6855 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
W20260808 01:50:49.783901 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d367581a54c433ea4f227bd4f479ed2. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:49.786584 6862 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.796175 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
W20260808 01:50:49.796757 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5843450fd1f54792899cc7190c94821f. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:49.797711 6855 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20260808 01:50:49.800007 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20260808 01:50:49.801534 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:49.797938 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.807588 4786 catalog_manager.cc:5697] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.813146 6862 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20260808 01:50:49.832307 6845 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.844358 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:49.859416 6658 consensus_peers.cc:313] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:49.865916 4770 catalog_manager.cc:5220] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 207 but the committed config has opid_index of 208
I20260808 01:50:49.871544 5074 consensus_queue.cc:237] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:49.880038 5205 raft_consensus.cc:1275] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20260808 01:50:49.881564 6853 consensus_queue.cc:1048] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20260808 01:50:49.889719 6693 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.891587 5213 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.891711 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.895025 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 205: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:49.891607 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 2.205, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:49.902177 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 3.206, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:49.905992 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 213cfcc36a754065ab9203bf5a8f6f65 with cas_config_opid_index 205: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:50:49.907543 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7aa70246b445149411741e595f27c2. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:49.908766 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 206. Preceding OpId from leader: term: 3 index: 207. (index mismatch)
I20260808 01:50:49.910986 6845 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20260808 01:50:49.893289 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.912876 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20260808 01:50:49.914541 6862 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20260808 01:50:49.922726 6867 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.923177 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.923712 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:49.925905 6866 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.207: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:50:49.934319 6658 consensus_peers.cc:313] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
I20260808 01:50:49.939791 4782 catalog_manager.cc:5697] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.953564 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:49.956867 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 205: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:50:49.960829 5010 consensus_peers.cc:597] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 213cfcc36a754065ab9203bf5a8f6f65. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:49.961812 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:49.980021 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 207, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:50.010713 5213 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 3.207, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:50:50.019649 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fff9439a0bd4286827363f69521c44f. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:50.023131 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 207. Preceding OpId from leader: term: 3 index: 208. (index mismatch)
I20260808 01:50:50.025359 6867 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:50.033982 6866 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.035956 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.052287 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.058099 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:50:50.218262 4818 debug-util.cc:398] Leaking SignalData structure 0x7b08000502a0 after lost signal to thread 4749
W20260808 01:50:50.219046 4818 debug-util.cc:398] Leaking SignalData structure 0x7b080009fc80 after lost signal to thread 4822
W20260808 01:50:50.309958 6866 consensus_peers.cc:313] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:50.344611 6866 consensus_peers.cc:313] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:50.348572 6866 consensus_peers.cc:313] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:50.402297 6866 consensus_peers.cc:313] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:50.426982 6866 consensus_peers.cc:313] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
I20260808 01:50:50.438277 6886 ts_tablet_manager.cc:933] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
I20260808 01:50:50.456861 6886 tablet_copy_client.cc:323] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:43479
I20260808 01:50:50.495565 5255 generic_service.cc:175] {username='slave'} at 127.0.0.1:33916 changed flags via RPC: env_inject_eio from '0' to '0.0'
I20260808 01:50:50.506582 5095 tablet_copy_service.cc:140] P ec65d57b7135430eb6695032355bd356: Received BeginTabletCopySession request for tablet 213cfcc36a754065ab9203bf5a8f6f65 from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:45687)
I20260808 01:50:50.507313 5095 tablet_copy_service.cc:161] P ec65d57b7135430eb6695032355bd356: Beginning new tablet copy session on tablet 213cfcc36a754065ab9203bf5a8f6f65 from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:45687: session id = c37035739bf347d49e5ed4b2f884adac-213cfcc36a754065ab9203bf5a8f6f65
I20260808 01:50:50.522859 5095 tablet_copy_source_session.cc:215] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: Tablet Copy: opened 5 blocks and 1 log segments
W20260808 01:50:50.528875 5212 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.000s sys 0.001s
W20260808 01:50:50.529219 5212 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s user 0.000s sys 0.001s
W20260808 01:50:50.539930 6866 consensus_peers.cc:313] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled
W20260808 01:50:50.542613 5206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.008s sys 0.005s
W20260808 01:50:50.542933 5206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.008s sys 0.005s
W20260808 01:50:50.543210 5255 generic_service.cc:139] {username='slave'} at 127.0.0.1:33916 forcing change of non-runtime-safe flag cfile_inject_corruption
I20260808 01:50:50.543603 5255 generic_service.cc:175] {username='slave'} at 127.0.0.1:33916 changed flags via RPC: cfile_inject_corruption from '0' to '0.0'
W20260808 01:50:50.550189 5211 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.003s sys 0.011s
W20260808 01:50:50.550503 5211 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.003s sys 0.011s
W20260808 01:50:50.551131 5255 generic_service.cc:139] {username='slave'} at 127.0.0.1:33916 forcing change of non-runtime-safe flag tablet_inject_kudu_2233
I20260808 01:50:50.550447 6891 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.206: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.551514 5255 generic_service.cc:175] {username='slave'} at 127.0.0.1:33916 changed flags via RPC: tablet_inject_kudu_2233 from '0' to '0.0'
W20260808 01:50:50.555433 5255 generic_service.cc:139] {username='slave'} at 127.0.0.1:33916 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:50.555905 5255 generic_service.cc:175] {username='slave'} at 127.0.0.1:33916 changed flags via RPC: enable_tablet_copy from 'false' to 'true'
W20260808 01:50:50.559731 6611 generic_service.cc:139] {username='slave'} at 127.0.0.1:41366 forcing change of non-runtime-safe flag env_inject_eio
I20260808 01:50:50.581045 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.206: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.583225 6893 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.587422 6892 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.587918 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.592309 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.205: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New config: { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.602566 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:50.619012 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:50.623339 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:50.625995 6886 tablet_copy_client.cc:806] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:50.629524 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:50.661995 6886 tablet_copy_client.cc:670] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:50.678828 6886 tablet_copy_client.cc:538] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:50.678959 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:50.688545 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 205, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 205 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.692149 6886 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:50.691834 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index -1 to 206, VOTER 01adc8a4bde24946b6bf7e18e876e977 (127.29.180.65) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:50.695376 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
W20260808 01:50:50.697266 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86d58fdfa95b45f68c0648a585924d05. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:50.697896 6893 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:50.705543 6892 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.715763 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.716576 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:50.722007 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:50.726567 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
W20260808 01:50:50.728626 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ee6f0c29217471e93394d4b9814144e. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:50.729928 6893 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20260808 01:50:50.736634 6853 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.738469 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.754241 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
W20260808 01:50:50.756106 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a97be89f618431b90b49c97b2c9f001. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:50.757501 6893 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:50.758642 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 205: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:50.760638 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 206: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:50.764197 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 205 to 206, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:50.764457 6853 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.776860 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 206: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:50.778285 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.783794 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:50.799160 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:51.220449 6886 log.cc:826] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Log is configured to *not* fsync() on all Append() calls
W20260808 01:50:52.147548 6556 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.396s user 0.002s sys 0.000s
W20260808 01:50:52.148020 6556 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.397s user 0.002s sys 0.000s
I20260808 01:50:52.148428 6900 ts_tablet_manager.cc:933] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:52.148816 6544 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.002s sys 0.000s
W20260808 01:50:52.149153 6544 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.838s user 0.003s sys 0.000s
W20260808 01:50:52.150709 6553 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.257s user 0.002s sys 0.000s
W20260808 01:50:52.151070 6553 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.257s user 0.002s sys 0.000s
I20260808 01:50:52.151484 6902 ts_tablet_manager.cc:933] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:52.153832 6552 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.223s user 0.004s sys 0.000s
W20260808 01:50:52.154241 6552 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.224s user 0.004s sys 0.001s
I20260808 01:50:52.154410 6904 ts_tablet_manager.cc:933] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:52.156915 6551 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.205s user 0.004s sys 0.000s
I20260808 01:50:52.157066 6910 ts_tablet_manager.cc:933] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
I20260808 01:50:52.157574 6905 ts_tablet_manager.cc:933] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:52.157719 6550 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.203s user 0.001s sys 0.002s
I20260808 01:50:52.157858 6611 generic_service.cc:175] {username='slave'} at 127.0.0.1:41366 changed flags via RPC: env_inject_eio from '0' to '0.0'
I20260808 01:50:52.158491 6906 ts_tablet_manager.cc:933] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
W20260808 01:50:52.172189 6886 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.010s sys 0.012s
W20260808 01:50:52.173143 6886 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.949s user 0.010s sys 0.012s
I20260808 01:50:52.173792 6902 tablet_copy_client.cc:323] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
W20260808 01:50:52.172266 6551 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.220s user 0.004s sys 0.000s
W20260808 01:50:52.172578 6550 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.218s user 0.001s sys 0.003s
W20260808 01:50:52.181746 6386 debug-util.cc:398] Leaking SignalData structure 0x7b0800028380 after lost signal to thread 6555
I20260808 01:50:52.182380 6901 ts_tablet_manager.cc:933] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:52.182477 6386 kernel_stack_watchdog.cc:198] Thread 6555 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 369ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260808 01:50:52.172783 6611 rpcz_store.cc:275] Call kudu.server.GenericService.SetFlag from 127.0.0.1:41366 (request call id 1) took 1613 ms. Trace:
I20260808 01:50:52.188731 6900 tablet_copy_client.cc:323] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
W20260808 01:50:52.188756 6547 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.002s sys 0.003s
W20260808 01:50:52.197870 6555 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.388s user 0.004s sys 0.006s
W20260808 01:50:52.203994 6555 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.404s user 0.004s sys 0.006s
W20260808 01:50:52.204660 6547 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.085s user 0.002s sys 0.003s
I20260808 01:50:52.188819 6611 rpcz_store.cc:276] 0808 01:50:50.558730 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.558951 (+ 221us) service_pool.cc:224] Handling call
0808 01:50:52.172424 (+1613473us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:52.207340 6546 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.958s user 0.003s sys 0.017s
I20260808 01:50:52.200299 6907 ts_tablet_manager.cc:933] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:52.208245 6546 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.959s user 0.004s sys 0.018s
I20260808 01:50:52.209583 6909 ts_tablet_manager.cc:933] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
W20260808 01:50:52.227191 6611 generic_service.cc:139] {username='slave'} at 127.0.0.1:41366 forcing change of non-runtime-safe flag cfile_inject_corruption
I20260808 01:50:52.227620 6611 generic_service.cc:175] {username='slave'} at 127.0.0.1:41366 changed flags via RPC: cfile_inject_corruption from '0' to '0.0'
W20260808 01:50:52.231714 6611 generic_service.cc:139] {username='slave'} at 127.0.0.1:41366 forcing change of non-runtime-safe flag tablet_inject_kudu_2233
I20260808 01:50:52.232235 6611 generic_service.cc:175] {username='slave'} at 127.0.0.1:41366 changed flags via RPC: tablet_inject_kudu_2233 from '0' to '0.0'
W20260808 01:50:52.235889 6611 generic_service.cc:139] {username='slave'} at 127.0.0.1:41366 forcing change of non-runtime-safe flag enable_tablet_copy
I20260808 01:50:52.236356 6611 generic_service.cc:175] {username='slave'} at 127.0.0.1:41366 changed flags via RPC: enable_tablet_copy from 'false' to 'true'
I20260808 01:50:52.245266 6904 tablet_copy_client.cc:323] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
I20260808 01:50:52.256654 6910 tablet_copy_client.cc:323] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
I20260808 01:50:52.258006 6907 tablet_copy_client.cc:323] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:43479
I20260808 01:50:52.259430 6909 tablet_copy_client.cc:323] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:43479
I20260808 01:50:52.262104 5095 tablet_copy_service.cc:140] P ec65d57b7135430eb6695032355bd356: Received BeginTabletCopySession request for tablet 9ee6f0c29217471e93394d4b9814144e from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:45687)
I20260808 01:50:52.262601 5095 tablet_copy_service.cc:161] P ec65d57b7135430eb6695032355bd356: Beginning new tablet copy session on tablet 9ee6f0c29217471e93394d4b9814144e from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:45687: session id = c37035739bf347d49e5ed4b2f884adac-9ee6f0c29217471e93394d4b9814144e
I20260808 01:50:52.271286 5095 tablet_copy_source_session.cc:215] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.278752 6906 tablet_copy_client.cc:323] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
I20260808 01:50:52.278899 6901 tablet_copy_client.cc:323] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.66:43479
I20260808 01:50:52.298141 5095 tablet_copy_service.cc:140] P ec65d57b7135430eb6695032355bd356: Received BeginTabletCopySession request for tablet 2a97be89f618431b90b49c97b2c9f001 from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:45687)
I20260808 01:50:52.298797 5095 tablet_copy_service.cc:161] P ec65d57b7135430eb6695032355bd356: Beginning new tablet copy session on tablet 2a97be89f618431b90b49c97b2c9f001 from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:45687: session id = c37035739bf347d49e5ed4b2f884adac-2a97be89f618431b90b49c97b2c9f001
I20260808 01:50:52.323035 6905 tablet_copy_client.cc:323] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Beginning tablet copy session from remote peer at address 127.29.180.67:46093
I20260808 01:50:52.325078 5233 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 8d367581a54c433ea4f227bd4f479ed2 from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.325894 5231 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 2d7aa70246b445149411741e595f27c2 from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.326493 5234 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 1fff9439a0bd4286827363f69521c44f from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.326658 5230 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 0c44eefb43024e9290fdba8647e745fe from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.327399 5229 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 5843450fd1f54792899cc7190c94821f from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.328346 5228 tablet_copy_service.cc:140] P 99d0ee4efbd948a3b143ff2297ad9fc9: Received BeginTabletCopySession request for tablet 24ad2932d9a1489b96a63bc762f6a23f from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:52015)
I20260808 01:50:52.328882 5228 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 24ad2932d9a1489b96a63bc762f6a23f from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-24ad2932d9a1489b96a63bc762f6a23f
I20260808 01:50:52.333927 5233 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 8d367581a54c433ea4f227bd4f479ed2 from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-8d367581a54c433ea4f227bd4f479ed2
I20260808 01:50:52.334892 5231 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 2d7aa70246b445149411741e595f27c2 from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-2d7aa70246b445149411741e595f27c2
I20260808 01:50:52.337220 5229 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 5843450fd1f54792899cc7190c94821f from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-5843450fd1f54792899cc7190c94821f
I20260808 01:50:52.338435 5234 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 1fff9439a0bd4286827363f69521c44f from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-1fff9439a0bd4286827363f69521c44f
I20260808 01:50:52.329531 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.66:51607 (request call id 6) took 1211 ms. Trace:
I20260808 01:50:52.339800 5230 tablet_copy_service.cc:161] P 99d0ee4efbd948a3b143ff2297ad9fc9: Beginning new tablet copy session on tablet 0c44eefb43024e9290fdba8647e745fe from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:52015: session id = c37035739bf347d49e5ed4b2f884adac-0c44eefb43024e9290fdba8647e745fe
I20260808 01:50:52.339636 6907 rpcz_store.cc:276] 0808 01:50:51.118310 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.118648 (+ 338us) service_pool.cc:224] Handling call
0808 01:50:52.208389 (+1089741us) ts_tablet_manager.cc:934] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
0808 01:50:52.281093 (+ 72704us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.295200 (+ 14107us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.295311 (+ 111us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.328787 (+ 33476us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.329084 (+ 297us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.329392 (+ 308us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.329459 (+ 67us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":285,"tablet-copy.queue_time_us":1054351,"thread_start_us":1129811,"threads_started":2}
I20260808 01:50:52.352715 5229 tablet_copy_source_session.cc:215] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.324620 5095 tablet_copy_source_session.cc:215] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.353807 6907 tablet_copy_client.cc:806] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.326186 5094 tablet_copy_service.cc:140] P ec65d57b7135430eb6695032355bd356: Received BeginTabletCopySession request for tablet 86d58fdfa95b45f68c0648a585924d05 from peer c37035739bf347d49e5ed4b2f884adac ({username='slave'} at 127.29.180.68:45687)
I20260808 01:50:52.356720 5094 tablet_copy_service.cc:161] P ec65d57b7135430eb6695032355bd356: Beginning new tablet copy session on tablet 86d58fdfa95b45f68c0648a585924d05 from peer c37035739bf347d49e5ed4b2f884adac at {username='slave'} at 127.29.180.68:45687: session id = c37035739bf347d49e5ed4b2f884adac-86d58fdfa95b45f68c0648a585924d05
I20260808 01:50:52.360357 5233 tablet_copy_source_session.cc:215] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.364095 5228 tablet_copy_source_session.cc:215] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.372604 5094 tablet_copy_source_session.cc:215] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.376940 5230 tablet_copy_source_session.cc:215] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.380012 5234 tablet_copy_source_session.cc:215] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.381876 5231 tablet_copy_source_session.cc:215] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9: Tablet Copy: opened 5 blocks and 1 log segments
I20260808 01:50:52.444834 6901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.66:51607 (request call id 7) took 1196 ms. Trace:
I20260808 01:50:52.514842 6907 tablet_copy_client.cc:670] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:52.445292 6901 rpcz_store.cc:276] 0808 01:50:51.248385 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.248740 (+ 355us) service_pool.cc:224] Handling call
0808 01:50:52.182645 (+933905us) ts_tablet_manager.cc:934] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
0808 01:50:52.407123 (+224478us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.418214 (+ 11091us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.418331 (+ 117us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.444036 (+ 25705us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.444343 (+ 307us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.444685 (+ 342us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.444755 (+ 70us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":6905,"tablet-copy.queue_time_us":928147,"thread_start_us":1028694,"threads_started":2}
I20260808 01:50:52.522919 6901 tablet_copy_client.cc:806] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.526113 6900 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 6) took 1775 ms. Trace:
I20260808 01:50:52.528132 6900 rpcz_store.cc:276] 0808 01:50:50.750537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.750703 (+ 166us) service_pool.cc:224] Handling call
0808 01:50:52.148831 (+1398128us) ts_tablet_manager.cc:934] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.365703 (+216872us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.481037 (+115334us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.481169 (+ 132us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.525377 (+ 44208us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.525683 (+ 306us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.525984 (+ 301us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.526043 (+ 59us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":17027,"tablet-copy.queue_time_us":1396727,"thread_start_us":1406892,"threads_started":2}
I20260808 01:50:52.552438 6900 tablet_copy_client.cc:806] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.535319 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 7) took 1735 ms. Trace:
I20260808 01:50:52.553341 6910 rpcz_store.cc:276] 0808 01:50:50.799908 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.800058 (+ 150us) service_pool.cc:224] Handling call
0808 01:50:52.173128 (+1373070us) ts_tablet_manager.cc:934] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.408100 (+234972us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.477503 (+ 69403us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.484960 (+ 7457us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.534618 (+ 49658us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.534885 (+ 267us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.535194 (+ 309us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.535254 (+ 60us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":405,"tablet-copy.queue_time_us":1347535,"thread_start_us":1436912,"threads_started":2}
I20260808 01:50:52.554829 6910 tablet_copy_client.cc:806] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.551844 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 8) took 1658 ms. Trace:
I20260808 01:50:52.532161 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 10) took 1581 ms. Trace:
I20260808 01:50:52.557221 6905 rpcz_store.cc:276] 0808 01:50:50.951129 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.951354 (+ 225us) service_pool.cc:224] Handling call
0808 01:50:52.173062 (+1221708us) ts_tablet_manager.cc:934] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.378438 (+205376us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.479711 (+101273us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.479854 (+ 143us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.531485 (+ 51631us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.531746 (+ 261us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.532046 (+ 300us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.532099 (+ 53us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":13220,"tablet-copy.queue_time_us":1205348,"thread_start_us":1342892,"threads_started":2}
I20260808 01:50:52.568231 6905 tablet_copy_client.cc:806] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.568976 6907 tablet_copy_client.cc:538] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.570639 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 11) took 1617 ms. Trace:
I20260808 01:50:52.571133 6906 rpcz_store.cc:276] 0808 01:50:50.953561 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.953803 (+ 242us) service_pool.cc:224] Handling call
0808 01:50:52.173891 (+1220088us) ts_tablet_manager.cc:934] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.380135 (+206244us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.465347 (+ 85212us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.465498 (+ 151us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.551066 (+ 85568us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.570160 (+ 19094us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.570508 (+ 348us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.570575 (+ 67us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":1521,"tablet-copy.queue_time_us":1203836,"thread_start_us":1299450,"threads_started":2}
I20260808 01:50:52.572659 6906 tablet_copy_client.cc:806] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.555418 6902 rpcz_store.cc:276] 0808 01:50:50.893062 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.893228 (+ 166us) service_pool.cc:224] Handling call
0808 01:50:52.151757 (+1258529us) ts_tablet_manager.cc:934] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.432626 (+280869us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.483867 (+ 51241us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.484004 (+ 137us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.514812 (+ 30808us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.515067 (+ 255us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.551675 (+ 36608us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.551771 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":1621,"tablet-copy.queue_time_us":1257492,"thread_start_us":1276714,"threads_started":2}
I20260808 01:50:52.573417 6902 tablet_copy_client.cc:806] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.581264 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.67:58773 (request call id 9) took 1651 ms. Trace:
I20260808 01:50:52.563474 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.180.66:51607 (request call id 8) took 1253 ms. Trace:
I20260808 01:50:52.587355 6909 rpcz_store.cc:276] 0808 01:50:51.310618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.310967 (+ 349us) service_pool.cc:224] Handling call
0808 01:50:52.209851 (+898884us) ts_tablet_manager.cc:934] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479)
0808 01:50:52.367151 (+157300us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.489119 (+121968us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.489225 (+ 106us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.562712 (+ 73487us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.563025 (+ 313us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.563339 (+ 314us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.563404 (+ 65us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":259,"tablet-copy.queue_time_us":897292,"thread_start_us":886399,"threads_started":2}
I20260808 01:50:52.588651 6909 tablet_copy_client.cc:806] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.586923 6904 rpcz_store.cc:276] 0808 01:50:50.929730 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.930057 (+ 327us) service_pool.cc:224] Handling call
0808 01:50:52.154713 (+1224656us) ts_tablet_manager.cc:934] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: Initiating tablet copy from peer 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093)
0808 01:50:52.467835 (+313122us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:52.521060 (+ 53225us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:52.521178 (+ 118us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:52.545557 (+ 24379us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:52.580777 (+ 35220us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:52.581116 (+ 339us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:52.581185 (+ 69us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":36696,"tablet-copy.queue_time_us":1223552,"thread_start_us":1257954,"threads_started":2}
I20260808 01:50:52.590250 6904 tablet_copy_client.cc:806] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 5 data blocks...
I20260808 01:50:52.602140 6907 tablet_bootstrap.cc:492] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:52.671597 6901 tablet_copy_client.cc:670] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
W20260808 01:50:52.745460 6919 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data (482619392 bytes available vs 518351011 bytes reserved) (error 28)
W20260808 01:50:52.754570 6916 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data (482611200 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:52.765384 6901 tablet_copy_client.cc:538] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.785096 6901 tablet_bootstrap.cc:492] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
W20260808 01:50:52.794569 6920 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data (448929792 bytes available vs 518351011 bytes reserved) (error 28)
W20260808 01:50:52.804584 6917 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/data (415363072 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:52.825558 6905 tablet_copy_client.cc:670] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:52.828096 6910 tablet_copy_client.cc:670] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
W20260808 01:50:52.842625 6917 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data (357277696 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:52.845579 6902 tablet_copy_client.cc:670] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:52.847932 6900 tablet_copy_client.cc:670] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
W20260808 01:50:52.862391 6920 dir_manager.cc:275] Insufficient disk space under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-1/data (348225536 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:52.880788 6905 tablet_copy_client.cc:538] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.893603 6905 tablet_bootstrap.cc:492] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:52.907357 6904 tablet_copy_client.cc:670] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:52.910805 6910 tablet_copy_client.cc:538] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.919070 6900 tablet_copy_client.cc:538] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.926812 6910 tablet_bootstrap.cc:492] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:52.929700 6900 tablet_bootstrap.cc:492] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:52.961696 6902 tablet_copy_client.cc:538] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:52.982750 6902 tablet_bootstrap.cc:492] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:52.994776 6909 tablet_copy_client.cc:670] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:53.011660 6906 tablet_copy_client.cc:670] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:53.021592 6904 tablet_copy_client.cc:538] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:53.059635 6909 tablet_copy_client.cc:538] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:53.061512 6904 tablet_bootstrap.cc:492] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:53.096907 6909 tablet_bootstrap.cc:492] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
I20260808 01:50:53.129029 6906 tablet_copy_client.cc:538] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260808 01:50:53.151404 5010 consensus_peers.cc:597] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:50:53.154175 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:53.162319 6906 tablet_bootstrap.cc:492] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: Bootstrap starting.
W20260808 01:50:53.306661 6901 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.309870 6901 ts_tablet_manager.cc:1425] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.310384 6901 ts_tablet_manager.cc:1403] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.526s user 0.148s sys 0.009s
I20260808 01:50:53.310778 6901 tablet_replica.cc:333] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.311183 6901 raft_consensus.cc:2243] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.311925 6901 raft_consensus.cc:2272] T 86d58fdfa95b45f68c0648a585924d05 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.314299 5093 tablet_copy_service.cc:342] P ec65d57b7135430eb6695032355bd356: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-86d58fdfa95b45f68c0648a585924d05 received from {username='slave'} at 127.29.180.68:45687
I20260808 01:50:53.314818 5093 tablet_copy_service.cc:434] P ec65d57b7135430eb6695032355bd356: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-86d58fdfa95b45f68c0648a585924d05 on tablet 86d58fdfa95b45f68c0648a585924d05 with peer c37035739bf347d49e5ed4b2f884adac
W20260808 01:50:53.340273 6886 log.cc:927] Time spent T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Append to log took a long time: real 0.054s user 0.018s sys 0.036s
I20260808 01:50:53.367260 6886 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=203} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:50:53.368489 6886 tablet_bootstrap.cc:492] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Bootstrap complete.
I20260808 01:50:53.369346 6886 ts_tablet_manager.cc:1403] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 2.678s user 0.764s sys 0.082s
W20260808 01:50:53.374038 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:53.389389 6886 raft_consensus.cc:359] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:53.390733 6886 raft_consensus.cc:740] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c37035739bf347d49e5ed4b2f884adac, State: Initialized, Role: LEARNER
I20260808 01:50:53.391594 6886 consensus_queue.cc:260] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:53.402819 6886 ts_tablet_manager.cc:1434] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac: Time spent starting tablet: real 0.033s user 0.022s sys 0.008s
I20260808 01:50:53.408509 5093 tablet_copy_service.cc:342] P ec65d57b7135430eb6695032355bd356: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-213cfcc36a754065ab9203bf5a8f6f65 received from {username='slave'} at 127.29.180.68:45687
I20260808 01:50:53.408993 5093 tablet_copy_service.cc:434] P ec65d57b7135430eb6695032355bd356: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-213cfcc36a754065ab9203bf5a8f6f65 on tablet 213cfcc36a754065ab9203bf5a8f6f65 with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.526608 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.541117 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:53.546449 6910 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.548465 6910 ts_tablet_manager.cc:1425] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.550134 6910 ts_tablet_manager.cc:1403] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.624s user 0.137s sys 0.019s
I20260808 01:50:53.550864 6910 tablet_replica.cc:333] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.551510 6910 raft_consensus.cc:2243] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.552400 6910 raft_consensus.cc:2272] T 0c44eefb43024e9290fdba8647e745fe P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.558089 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-0c44eefb43024e9290fdba8647e745fe received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:53.558624 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-0c44eefb43024e9290fdba8647e745fe on tablet 0c44eefb43024e9290fdba8647e745fe with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.574734 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20260808 01:50:53.583050 6546 raft_consensus.cc:1217] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.205->[1.206-1.206] Dedup: 1.206->[]
I20260808 01:50:53.593990 6912 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:53.609874 6621 maintenance_manager.cc:419] P c37035739bf347d49e5ed4b2f884adac: Scheduling UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65): 9643 bytes on disk
I20260808 01:50:53.612869 6476 maintenance_manager.cc:643] P c37035739bf347d49e5ed4b2f884adac: UndoDeltaBlockGCOp(213cfcc36a754065ab9203bf5a8f6f65) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":419,"lbm_reads_lt_1ms":4}
I20260808 01:50:53.623796 6912 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.631609 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.657426 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.675125 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:53.696707 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:53.697335 6907 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.699266 6907 ts_tablet_manager.cc:1425] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.699712 6907 ts_tablet_manager.cc:1403] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 1.100s user 0.165s sys 0.003s
I20260808 01:50:53.700066 6907 tablet_replica.cc:333] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.700357 6907 raft_consensus.cc:2243] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.700753 6907 raft_consensus.cc:2272] T 9ee6f0c29217471e93394d4b9814144e P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.703588 5093 tablet_copy_service.cc:342] P ec65d57b7135430eb6695032355bd356: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-9ee6f0c29217471e93394d4b9814144e received from {username='slave'} at 127.29.180.68:45687
I20260808 01:50:53.704020 5093 tablet_copy_service.cc:434] P ec65d57b7135430eb6695032355bd356: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-9ee6f0c29217471e93394d4b9814144e on tablet 9ee6f0c29217471e93394d4b9814144e with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.730031 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:53.737571 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20260808 01:50:53.737843 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:53.738375 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:53.739583 6912 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.001s
I20260808 01:50:53.741257 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:53.750425 6900 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
I20260808 01:50:53.750346 6934 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
E20260808 01:50:53.752278 6900 ts_tablet_manager.cc:1425] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.752584 6900 ts_tablet_manager.cc:1403] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.823s user 0.162s sys 0.000s
I20260808 01:50:53.752836 6900 tablet_replica.cc:333] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.753057 6900 raft_consensus.cc:2243] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.753342 6900 raft_consensus.cc:2272] T 8d367581a54c433ea4f227bd4f479ed2 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.755290 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-8d367581a54c433ea4f227bd4f479ed2 received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:53.755717 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-8d367581a54c433ea4f227bd4f479ed2 on tablet 8d367581a54c433ea4f227bd4f479ed2 with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.757771 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:53.794152 6909 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.796274 6909 ts_tablet_manager.cc:1425] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.796949 6909 ts_tablet_manager.cc:1403] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.700s user 0.166s sys 0.010s
I20260808 01:50:53.797319 6909 tablet_replica.cc:333] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.797649 6909 raft_consensus.cc:2243] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.798062 6909 raft_consensus.cc:2272] T 2a97be89f618431b90b49c97b2c9f001 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.799873 5093 tablet_copy_service.cc:342] P ec65d57b7135430eb6695032355bd356: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-2a97be89f618431b90b49c97b2c9f001 received from {username='slave'} at 127.29.180.68:45687
I20260808 01:50:53.800360 5093 tablet_copy_service.cc:434] P ec65d57b7135430eb6695032355bd356: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-2a97be89f618431b90b49c97b2c9f001 on tablet 2a97be89f618431b90b49c97b2c9f001 with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.801700 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:53.805667 6904 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.808403 6904 ts_tablet_manager.cc:1425] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.808689 6904 ts_tablet_manager.cc:1403] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.747s user 0.150s sys 0.026s
I20260808 01:50:53.808917 6904 tablet_replica.cc:333] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.809114 6904 raft_consensus.cc:2243] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.809414 6904 raft_consensus.cc:2272] T 2d7aa70246b445149411741e595f27c2 P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.812325 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-2d7aa70246b445149411741e595f27c2 received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:53.812783 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-2d7aa70246b445149411741e595f27c2 on tablet 2d7aa70246b445149411741e595f27c2 with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.816566 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 208: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:53.821262 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20260808 01:50:53.822942 6936 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.001s
I20260808 01:50:53.827055 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.828938 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:53.846606 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:53.854625 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20260808 01:50:53.856513 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:53.863855 6912 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.872184 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.889411 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20260808 01:50:53.891419 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 209: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:53.892292 6912 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20260808 01:50:53.898922 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:50:53.906786 6905 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:53.908618 6905 ts_tablet_manager.cc:1425] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:53.908921 6905 ts_tablet_manager.cc:1403] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 1.016s user 0.142s sys 0.022s
I20260808 01:50:53.909183 6905 tablet_replica.cc:333] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:53.909397 6905 raft_consensus.cc:2243] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:50:53.909713 6905 raft_consensus.cc:2272] T 24ad2932d9a1489b96a63bc762f6a23f P c37035739bf347d49e5ed4b2f884adac [term 2 LEARNER]: Raft consensus is shut down!
I20260808 01:50:53.911176 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-24ad2932d9a1489b96a63bc762f6a23f received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:53.911563 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-24ad2932d9a1489b96a63bc762f6a23f on tablet 24ad2932d9a1489b96a63bc762f6a23f with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:53.917995 6937 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.920428 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 2.210, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:53.925457 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.928078 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.928283 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 210. Preceding OpId from leader: term: 2 index: 211. (index mismatch)
W20260808 01:50:53.928510 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:53.929723 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:53.934746 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.945081 6912 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:53.949918 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:53.953999 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.958961 6935 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.961733 5213 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:53.970443 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:53.973520 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:53.981468 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:54.001044 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.024801 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 210: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.020902 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.027140 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.030102 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.037954 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20260808 01:50:54.039728 6935 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:54.047603 6912 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.059708 5213 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.060671 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 211: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.069569 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.073866 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:54.083452 6906 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
I20260808 01:50:54.084074 5213 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.085960 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:54.086349 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
E20260808 01:50:54.091081 6906 ts_tablet_manager.cc:1425] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:54.091403 6906 ts_tablet_manager.cc:1403] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 0.929s user 0.160s sys 0.019s
I20260808 01:50:54.091673 6906 tablet_replica.cc:333] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:54.091917 6906 raft_consensus.cc:2243] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus shutting down.
I20260808 01:50:54.092231 6906 raft_consensus.cc:2272] T 5843450fd1f54792899cc7190c94821f P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Raft consensus is shut down!
I20260808 01:50:54.095005 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-5843450fd1f54792899cc7190c94821f received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:54.095458 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-5843450fd1f54792899cc7190c94821f on tablet 5843450fd1f54792899cc7190c94821f with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:54.102221 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.106076 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.111075 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20260808 01:50:54.113888 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.114292 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.114087 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.001s
I20260808 01:50:54.117847 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20260808 01:50:54.126737 6937 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:54.134207 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.135803 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.143095 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.143567 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.143967 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20260808 01:50:54.144901 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:54.145526 6936 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.149848 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.001s
I20260808 01:50:54.156188 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.158896 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.158215 6912 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.163094 5213 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:54.166296 6952 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.29.180.126:32885, user_credentials={real_user=slave}} blocked reactor thread for 66560.3us
I20260808 01:50:54.191502 6972 raft_consensus.cc:1064] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356: attempting to promote NON_VOTER c37035739bf347d49e5ed4b2f884adac to VOTER
I20260808 01:50:54.194423 6972 consensus_queue.cc:237] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false } }
I20260808 01:50:54.201736 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20260808 01:50:54.203657 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.208400 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.213204 6546 raft_consensus.cc:1275] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [term 1 LEARNER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20260808 01:50:54.215886 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 212: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.218096 6972 consensus_queue.cc:1048] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:54.220942 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.221349 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:54.233199 6902 log.cc:1245] Error closing log: Illegal state: Log not open. State: 0
E20260808 01:50:54.235550 6902 ts_tablet_manager.cc:1425] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
I20260808 01:50:54.235949 6902 ts_tablet_manager.cc:1403] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: Time spent bootstrapping tablet: real 1.253s user 0.166s sys 0.004s
I20260808 01:50:54.236243 6902 tablet_replica.cc:333] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac: stopping tablet replica
I20260808 01:50:54.236560 6902 raft_consensus.cc:2243] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac [term 3 LEARNER]: Raft consensus shutting down.
I20260808 01:50:54.236987 6902 raft_consensus.cc:2272] T 1fff9439a0bd4286827363f69521c44f P c37035739bf347d49e5ed4b2f884adac [term 3 LEARNER]: Raft consensus is shut down!
I20260808 01:50:54.239177 6972 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 206 to 207, c37035739bf347d49e5ed4b2f884adac (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false } } }
I20260808 01:50:54.248900 6546 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P c37035739bf347d49e5ed4b2f884adac [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 206 to 207, c37035739bf347d49e5ed4b2f884adac (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false } } }
I20260808 01:50:54.251734 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:54.256793 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.258716 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.296236 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.313920 4782 catalog_manager.cc:5697] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 206 to 207, c37035739bf347d49e5ed4b2f884adac (127.29.180.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:54.319203 5231 tablet_copy_service.cc:342] P 99d0ee4efbd948a3b143ff2297ad9fc9: Request end of tablet copy session c37035739bf347d49e5ed4b2f884adac-1fff9439a0bd4286827363f69521c44f received from {username='slave'} at 127.29.180.68:52015
I20260808 01:50:54.319888 5231 tablet_copy_service.cc:434] P 99d0ee4efbd948a3b143ff2297ad9fc9: ending tablet copy session c37035739bf347d49e5ed4b2f884adac-1fff9439a0bd4286827363f69521c44f on tablet 1fff9439a0bd4286827363f69521c44f with peer c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:54.322139 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20260808 01:50:54.322031 6896 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20260808 01:50:54.324856 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:54.327721 5211 raft_consensus.cc:1275] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20260808 01:50:54.346627 5213 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20260808 01:50:54.350900 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:54.354653 6912 consensus_queue.cc:1048] T 213cfcc36a754065ab9203bf5a8f6f65 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20260808 01:50:54.356635 6912 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20260808 01:50:54.358215 6937 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.369089 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.382422 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.383011 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:50:54.383279 5213 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20260808 01:50:54.362839 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.393151 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.394678 6912 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:54.395812 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.398849 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 206: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.403126 5213 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.424041 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.436445 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20260808 01:50:54.440299 5206 raft_consensus.cc:2955] T 213cfcc36a754065ab9203bf5a8f6f65 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 206 to 207, c37035739bf347d49e5ed4b2f884adac (127.29.180.68) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: false } } }
I20260808 01:50:54.443991 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 2.210, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.452194 6937 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.001s
I20260808 01:50:54.454332 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:54.457075 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.468495 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 208: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.469101 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 210. Preceding OpId from leader: term: 2 index: 211. (index mismatch)
I20260808 01:50:54.471143 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.465062 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.476681 6937 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.001s
I20260808 01:50:54.479605 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:54.492120 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20260808 01:50:54.494860 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20260808 01:50:54.495177 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.492883 6936 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.481263 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:54.519057 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.524793 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.532421 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.532866 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.532938 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 3.208, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.540659 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20260808 01:50:54.542802 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:54.544610 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:50:54.544799 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.545224 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.547824 6896 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.553035 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.564182 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.566789 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.571051 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.584627 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.590907 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20260808 01:50:54.593794 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 208. Preceding OpId from leader: term: 3 index: 209. (index mismatch)
I20260808 01:50:54.595585 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.001s
I20260808 01:50:54.597656 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20260808 01:50:54.610157 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.635443 6937 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.635994 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.638799 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 1.209, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:54.650030 6997 log.cc:927] Time spent T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260808 01:50:54.641911 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.654201 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 209. Preceding OpId from leader: term: 1 index: 210. (index mismatch)
I20260808 01:50:54.655995 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.001s
I20260808 01:50:54.660889 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.660099 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.665735 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 206: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.667161 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.668990 6896 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.672297 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.673419 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 206: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.676002 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.679451 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.683804 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.684316 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.687001 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20260808 01:50:54.690193 6912 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:54.696537 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 209: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.698609 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.699935 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.699254 6934 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.710443 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.725997 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.727758 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.729481 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:50:54.739926 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.728986 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.762742 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 214: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.777554 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.792804 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 2.210, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.806274 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 210. Preceding OpId from leader: term: 2 index: 211. (index mismatch)
I20260808 01:50:54.808144 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:54.816160 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.818222 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.826236 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.830384 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:54.832309 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:54.833755 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.835503 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 206 to 207, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.836678 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:54.863473 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.869266 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20260808 01:50:54.871517 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.871922 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:54.877940 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.883884 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:54.886520 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.886956 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.890188 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.896166 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20260808 01:50:54.897921 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20260808 01:50:54.896026 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.902922 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20260808 01:50:54.899605 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20260808 01:50:54.909888 6934 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.912874 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.919054 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.926769 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.927186 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:54.927237 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 1 index: 211. (index mismatch)
I20260808 01:50:54.928670 6912 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:54.928668 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.934176 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.938555 6912 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.951406 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:54.963827 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 215: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:54.972258 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.975718 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 210: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:54.978561 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:54.980067 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:54.984128 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.989503 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:54.995869 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:54.996259 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.005405 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.013473 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.014775 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.015081 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.016573 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.018306 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.019318 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20260808 01:50:55.020524 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20260808 01:50:55.021097 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:55.022401 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.001s
I20260808 01:50:55.027877 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.028236 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.029780 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20260808 01:50:55.032785 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20260808 01:50:55.033031 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 3.209, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.035151 6937 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.037930 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.050676 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.051623 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.053571 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20260808 01:50:55.053845 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.054215 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.054802 6934 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.055336 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.058835 6936 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.059841 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.069084 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.069849 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 207: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.070188 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.072674 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.087572 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 209. Preceding OpId from leader: term: 3 index: 210. (index mismatch)
I20260808 01:50:55.090412 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.094434 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:55.016736 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.108989 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.119976 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.128427 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.145183 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.150197 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.155352 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.211, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.160090 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20260808 01:50:55.162716 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 212. (index mismatch)
I20260808 01:50:55.163086 6937 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.163679 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.164031 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.164204 6934 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.164592 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.164898 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.166512 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20260808 01:50:55.171461 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:55.172183 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 2.216, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:50:55.173332 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 208 but the committed config has opid_index of 209
I20260808 01:50:55.179057 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.179455 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.182085 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.183684 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.181118 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.187429 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.196483 6937 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.201874 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.208812 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 211: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.208071 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.212980 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 207 to 208, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.214613 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.211370 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 216. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20260808 01:50:55.216101 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.217325 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.002s
I20260808 01:50:55.225905 6934 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.231986 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.217453 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.238758 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.246819 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 216: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.254583 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.281073 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 1.209, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.289913 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.292913 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 209. Preceding OpId from leader: term: 1 index: 210. (index mismatch)
I20260808 01:50:55.293912 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.295276 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.298415 6934 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:55.301790 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20260808 01:50:55.300313 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.305312 6937 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20260808 01:50:55.308561 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 2.210, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.309100 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20260808 01:50:55.310904 6937 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.313334 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.313889 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.316491 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.318019 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.327524 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 3.210, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.329470 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 210. Preceding OpId from leader: term: 2 index: 211. (index mismatch)
I20260808 01:50:55.324249 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.335903 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.335866 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.331000 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:55.333868 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.209: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.341241 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.338879 6936 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.348450 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.350176 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.358736 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.376338 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 210. Preceding OpId from leader: term: 3 index: 211. (index mismatch)
I20260808 01:50:55.376919 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.378021 6936 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:55.382992 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.384292 6937 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.390951 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.391881 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.422498 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.428203 6827 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:54918 (request call id 5) took 1092 ms. Trace:
I20260808 01:50:55.433432 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 2.217, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.435583 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.438593 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.439455 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 2.210, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.442149 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 217. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20260808 01:50:55.444937 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:55.445988 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.453459 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.453874 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.457541 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20260808 01:50:55.457021 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.460700 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 210. Preceding OpId from leader: term: 2 index: 211. (index mismatch)
I20260808 01:50:55.462766 6934 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.465744 6827 rpcz_store.cc:276] 0808 01:50:54.336099 (+ 0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.336255 (+ 156us) service_pool.cc:224] Handling call
0808 01:50:55.428121 (+1091866us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:55.466557 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20260808 01:50:55.466428 6896 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:55.467871 6936 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.476845 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.483206 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.484390 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.485836 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.494372 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.495083 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.507097 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.514098 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.515306 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 217: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.514115 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 208 to 209, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.518026 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.520001 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 212: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.520931 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.523176 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.534129 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.545338 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.550845 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 1 index: 211. (index mismatch)
I20260808 01:50:55.552673 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:55.553221 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 1.209, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.559464 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.559679 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.560879 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.561213 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.562366 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.563383 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.568538 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 209. Preceding OpId from leader: term: 1 index: 210. (index mismatch)
I20260808 01:50:55.569693 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.573501 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.574925 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20260808 01:50:55.578088 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.578456 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.583736 6937 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.585350 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.210: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.599771 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:55.599781 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
W20260808 01:50:55.602360 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.603612 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 209: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.604350 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.607661 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 3.211, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.610345 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.612830 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.623634 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.624024 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.624054 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.629863 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.213, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.629809 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 211. Preceding OpId from leader: term: 3 index: 212. (index mismatch)
I20260808 01:50:55.634747 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.636674 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20260808 01:50:55.638006 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:55.638381 6896 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:55.640690 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20260808 01:50:55.642112 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
W20260808 01:50:55.646417 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.648414 6896 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.652837 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 2.218, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.659782 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.660451 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.660835 6936 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.663630 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.664023 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.667506 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 209 to 210, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.668769 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.670233 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.672215 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20260808 01:50:55.673758 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:55.681026 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.680637 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.684458 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.688618 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.698033 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.707911 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.709671 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.718061 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.723762 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 218: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.732432 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.735009 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.742172 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.211, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.748653 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.749046 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.752038 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 212. (index mismatch)
I20260808 01:50:55.753463 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.760130 6934 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.769960 5213 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.771421 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.775713 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.776310 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 3.212, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.776144 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 2.211, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.780541 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.780934 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.767920 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.782765 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20260808 01:50:55.783097 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20260808 01:50:55.784590 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20260808 01:50:55.785753 6937 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:55.787127 6937 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.789947 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 212. Preceding OpId from leader: term: 3 index: 213. (index mismatch)
I20260808 01:50:55.791863 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 211. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20260808 01:50:55.791724 6936 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:55.796460 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.797968 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.800172 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.800889 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.802326 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.802706 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I20260808 01:50:55.805840 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.809398 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.813974 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.824858 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 1 index: 211. (index mismatch)
I20260808 01:50:55.824604 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.827991 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.827903 6936 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.836678 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20260808 01:50:55.835858 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.838884 6937 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.846429 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.847589 7005 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.852440 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Master Summary
I20260808 01:50:55.858099 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.861554 7006 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.863564 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 1.214, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:55.858733 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.871188 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:55.871631 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:55.873056 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:55.872870 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20260808 01:50:55.874617 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 214. Preceding OpId from leader: term: 1 index: 215. (index mismatch)
I20260808 01:50:55.876590 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:55.878072 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:55.878372 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.882781 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.884157 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.886327 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.211: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.886500 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.889906 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 210: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.891736 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.895205 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
I20260808 01:50:55.897712 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:50:55.901942 6934 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:55.909286 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 207 | Yes
B | A B* C~ | 2 | 208 | Yes
C | [config not available] | | |
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 1 | 207 | No
B | A* B | 1 | 207 | Yes
C | [config not available] | | |
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 3 | 208 | Yes
B | A B* | 3 | 209 | Yes
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
I20260808 01:50:55.922435 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 2 | 214 | Yes
B | A* B C~ | 2 | 215 | No
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 1 | 207 | No
B | A B* | 1 | 207 | Yes
C | [config not available] | | |
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:55.931408 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 209 | Yes
B | A B* | 2 | 209 | Yes
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:50:55.938694 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:55.946023 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 219: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.946813 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 214: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 1 | 209 | Yes
B | A* B | 1 | 210 | Yes
C | [config not available] | | |
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:55.956117 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 211 | Yes
B | A B* | 2 | 211 | Yes
C | [config not available] | | |
Tablet 213cfcc36a754065ab9203bf5a8f6f65 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): RUNNING [NONVOTER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C | 1 | 207 | Yes
B | A* B C | 1 | 207 | Yes
C | A* B C | 1 | 207 | Yes
The cluster doesn't have any matching system tables
Summary by table
I20260808 01:50:55.959877 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 1 | 0 | 0 | 9
I20260808 01:50:55.960780 6936 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 30
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:50:55.971776 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:50:55.977897 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:55.979642 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:55.981135 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20260808 01:50:55.980947 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 210 to 211, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.983012 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.983508 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:55.984115 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.985157 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:55.985754 6937 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.988390 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.991151 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:55.998610 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:55.999105 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.004426 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.007994 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.012379 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.016294 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.211, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.016829 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 3.213, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.016948 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.022420 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.022931 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.023509 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.026225 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 211. Preceding OpId from leader: term: 1 index: 212. (index mismatch)
W20260808 01:50:56.029580 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.030819 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.030978 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20260808 01:50:56.031025 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
W20260808 01:50:56.031153 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.030588 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.032588 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 213. Preceding OpId from leader: term: 3 index: 214. (index mismatch)
I20260808 01:50:56.034091 7008 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.001s
I20260808 01:50:56.035138 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.035741 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.036125 7008 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.036499 6937 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:56.037307 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.001s
I20260808 01:50:56.034097 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20260808 01:50:56.038475 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.038805 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.039332 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.039937 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20260808 01:50:56.045542 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.213, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.048642 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.049255 7006 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.051277 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.052845 6937 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.058121 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.059890 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.060292 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.065892 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.066272 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.067278 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20260808 01:50:56.071887 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20260808 01:50:56.070055 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.072917 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.077930 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:56.079185 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 215: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.084589 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:56.086324 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.086573 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.086386 7008 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.091349 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.092875 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.094399 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.212: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.098078 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 211: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.098871 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.098280 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.100378 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.105847 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.106896 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.213, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.112277 6937 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.113924 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20260808 01:50:56.115716 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.117524 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:56.121193 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.121883 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:56.127166 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.128005 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 220: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.128713 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.133522 6934 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.144472 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.146220 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 211 to 212, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 212 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.147948 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.149868 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.151473 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.151679 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.152905 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.155047 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.178979 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.179497 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.183065 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20260808 01:50:56.184594 6896 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.186295 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20260808 01:50:56.185869 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.187707 6896 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20260808 01:50:56.193601 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.193993 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.194413 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20260808 01:50:56.196936 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.197824 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.198330 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.196936 6937 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.207657 7005 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.212353 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.212631 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.222704 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.231607 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.239773 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.241245 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.243932 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.245573 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.256927 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.259995 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 212: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.270097 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.278476 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.285223 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.285609 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.288858 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20260808 01:50:56.290560 6934 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.292176 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 2.216, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.297122 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20260808 01:50:56.299676 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:56.293694 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.304054 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 212 to 213, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.305210 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 3.214, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.305191 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.306062 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.311260 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 214. Preceding OpId from leader: term: 3 index: 215. (index mismatch)
I20260808 01:50:56.312629 6937 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.331580 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 1.214, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.335459 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.336259 6937 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.339303 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 216. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20260808 01:50:56.341089 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.343739 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.347488 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.349251 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.350700 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 216: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.351454 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.353709 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.359050 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.213, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.352027 6934 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.368340 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 2.216, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.361352 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20260808 01:50:56.370217 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20260808 01:50:56.373286 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.374296 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.383467 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 221: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:50:56.384171 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.385180 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.385870 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:50:56.386682 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.387465 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.388937 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch)
I20260808 01:50:56.388741 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.392072 7005 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.392671 6937 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.394115 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 216. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20260808 01:50:56.395177 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I20260808 01:50:56.395483 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 214. Preceding OpId from leader: term: 1 index: 215. (index mismatch)
I20260808 01:50:56.398510 7008 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.401800 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.405647 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.407378 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.416026 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 3.215, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.417680 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.420158 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.422725 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.425997 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 215. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20260808 01:50:56.426306 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.427979 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.431546 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.431919 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.431928 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20260808 01:50:56.433975 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.434449 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.435046 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:56.435722 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.438167 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.436700 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20260808 01:50:56.443104 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.445312 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.439608 7008 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.450970 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.454135 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 217: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.458377 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:56.461196 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.469605 7005 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.470362 7006 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.473470 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.481258 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.481594 7007 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.483582 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.486773 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.488214 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.491066 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.491611 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 2.217, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.493422 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.495270 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.499830 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.502105 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 213: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.502795 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.504112 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.506640 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.506810 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.511647 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.512060 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.514742 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 217. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20260808 01:50:56.519223 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:56.525772 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.530419 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.531924 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.532274 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.539059 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.547238 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.549189 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 213 to 214, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.550439 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.550590 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.557826 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 2.217, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.562005 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 2.216, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.566598 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 222: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.567817 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20260808 01:50:56.572300 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.572692 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.572749 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:56.574385 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 217. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20260808 01:50:56.576148 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:56.578933 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.585263 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:56.585330 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 216. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
W20260808 01:50:56.585638 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.579644 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.588212 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.586892 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.589753 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20260808 01:50:56.589851 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.591113 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.590955 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.583179 6896 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.599242 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.601269 6934 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.604840 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.601146 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.609460 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.610615 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 218: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.612129 7007 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.621591 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.639322 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.641202 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.641377 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 3.216, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.645756 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 1.214, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.646955 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 2.223, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.648010 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 216. Preceding OpId from leader: term: 3 index: 217. (index mismatch)
I20260808 01:50:56.649956 6896 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.649292 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.652104 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 214. Preceding OpId from leader: term: 1 index: 215. (index mismatch)
I20260808 01:50:56.653278 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.657399 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.215, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.661774 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 2.218, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.662750 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20260808 01:50:56.664388 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.664757 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.666512 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 223. Preceding OpId from leader: term: 2 index: 224. (index mismatch)
I20260808 01:50:56.667886 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:56.669081 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.669871 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.674846 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20260808 01:50:56.675184 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.677372 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20260808 01:50:56.679623 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.683215 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.683827 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.684767 6896 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:56.688930 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 223: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.695861 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.697556 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.708379 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 214: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.714201 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.215: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.715212 7007 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.724424 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.733983 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.734508 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.733057 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.738945 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 2.218, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.744156 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.744869 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 214 to 215, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.745018 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.746379 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.746290 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.747131 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.747864 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.752068 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20260808 01:50:56.753587 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:56.763512 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20260808 01:50:56.765854 6934 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.766769 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:56.772096 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20260808 01:50:56.774725 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:56.775570 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.777510 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.782714 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.785007 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 2.224, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.787391 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.791528 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.792028 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.795574 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 224. Preceding OpId from leader: term: 2 index: 225. (index mismatch)
I20260808 01:50:56.795459 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.802395 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.001s
I20260808 01:50:56.802925 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 2.217, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.801790 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.810248 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.812862 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 217. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20260808 01:50:56.815620 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.816196 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.816612 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:56.807001 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.825008 7007 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.827612 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.829885 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.818229 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.837607 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.852540 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.856178 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 224: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.857756 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 219: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.868534 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.884230 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:56.886919 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.889140 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:56.894528 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.918685 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.926944 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20260808 01:50:56.927910 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.928301 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.928509 7005 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20260808 01:50:56.935529 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 3.217, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.931016 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 2.216, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.939714 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.943921 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 216. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20260808 01:50:56.943828 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.946038 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.946532 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.949568 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:56.949853 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.006s
I20260808 01:50:56.961401 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20260808 01:50:56.963626 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:56.964257 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.967003 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:56.967576 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:56.967640 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:56.972747 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 217. Preceding OpId from leader: term: 3 index: 218. (index mismatch)
I20260808 01:50:56.974648 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:56.977458 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.988183 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:56.988261 6896 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:56.991123 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 215: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:56.995929 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 2.225, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:56.996789 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.001585 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.001806 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.008486 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 1.220, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.012759 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.013120 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.014274 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 225. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20260808 01:50:57.014958 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.016031 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:57.019523 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 215 to 216, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.026270 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20260808 01:50:57.027766 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:57.021462 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.029254 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20260808 01:50:57.033490 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.033880 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.036548 6934 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.038506 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.050004 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 225: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.049528 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.054349 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.056404 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.057657 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.058002 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.058141 7017 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.061149 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.071259 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 220: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.061149 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.081180 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.083824 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.089956 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch)
I20260808 01:50:57.094235 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.095180 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 2.217, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.095494 6896 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20260808 01:50:57.097429 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.098471 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 3.218, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.098505 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.099252 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.105928 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20260808 01:50:57.106794 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.098593 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.107543 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 2.218, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:57.111078 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.113456 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:57.114110 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20260808 01:50:57.115361 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20260808 01:50:57.115768 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 217. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20260808 01:50:57.115217 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.116745 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.117116 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.119316 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:57.120656 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.121965 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:57.122247 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.123461 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 218. Preceding OpId from leader: term: 3 index: 219. (index mismatch)
I20260808 01:50:57.128084 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.130506 6896 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:57.131448 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.133582 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.139384 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.142602 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.144677 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.145017 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.145071 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.147478 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20260808 01:50:57.148839 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:57.149469 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.152019 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.154539 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.154843 7005 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.157559 7017 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.159360 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.163764 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.163962 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.166843 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.174213 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.175536 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.180009 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20260808 01:50:57.181824 7017 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.001s
I20260808 01:50:57.186373 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.187165 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.189813 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.202646 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 221: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.203028 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.206267 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.213764 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.215407 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.217046 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.219573 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 216 to 217, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.222040 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 2 index: 227. (index mismatch)
I20260808 01:50:57.222036 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.223713 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:57.225059 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:50:57.232520 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 218 but the committed config has opid_index of 219
W20260808 01:50:57.233491 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 220 but the committed config has opid_index of 221
I20260808 01:50:57.245568 7017 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.253710 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.264941 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 226: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.268116 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:57.267289 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.276544 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.292820 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:50:57.296130 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20260808 01:50:57.297734 7017 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:57.270062 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.303911 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.308388 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 1.222, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.311725 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:50:57.314667 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.322077 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.332424 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20260808 01:50:57.333884 7017 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:57.333995 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 3.219, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.340989 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 219. Preceding OpId from leader: term: 3 index: 220. (index mismatch)
I20260808 01:50:57.340791 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.342588 7007 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:57.342171 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.331703 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 2.218, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.349732 7005 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.351284 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.351395 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 222: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:50:57.351663 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.355075 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.355432 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.354218 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.357156 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.357784 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.361212 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.362780 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20260808 01:50:57.363656 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20260808 01:50:57.364197 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:57.365935 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.366479 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.366873 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.368620 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20260808 01:50:57.373948 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.374297 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.374476 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20260808 01:50:57.375267 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.376003 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.376848 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.377187 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.378285 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20260808 01:50:57.378742 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.380452 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.380563 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.382658 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20260808 01:50:57.382869 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.384182 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:57.385150 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.390529 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.390964 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.402884 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.408833 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.412446 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:57.418149 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.422864 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 218. Preceding OpId from leader: term: 2 index: 219. (index mismatch)
I20260808 01:50:57.428203 7007 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:57.431730 7042 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.437063 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.450103 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.454008 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.462476 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.464138 7005 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.466988 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.471918 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 217: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.488731 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.490628 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.494792 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 217 to 218, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.496090 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.508419 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.520267 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.521842 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.537987 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.543298 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20260808 01:50:57.544862 7005 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.555768 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.557683 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.564360 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.567791 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.573563 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 1.223, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.577257 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20260808 01:50:57.578879 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20260808 01:50:57.578727 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:57.580965 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:57.584959 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.588155 7042 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.591089 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20260808 01:50:57.592561 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20260808 01:50:57.604579 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.605041 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.609109 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.613379 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.607043 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.615131 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.615231 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.617412 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.619655 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.623487 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 1.220, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.624162 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.626518 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.629364 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20260808 01:50:57.631009 6934 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.636234 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20260808 01:50:57.638564 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:57.639951 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.632560 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 218: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.645493 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.647099 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 223: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.650025 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.655510 7017 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.657485 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.668646 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 227: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.685485 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 2.223, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.687182 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.692335 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.692595 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.692875 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 3.220, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.696662 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 223. Preceding OpId from leader: term: 2 index: 224. (index mismatch)
I20260808 01:50:57.698355 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 220. Preceding OpId from leader: term: 3 index: 221. (index mismatch)
I20260808 01:50:57.700193 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.700574 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.701092 7042 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:57.701161 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.705395 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20260808 01:50:57.707170 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:57.707599 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.712442 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
W20260808 01:50:57.712805 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.713474 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.715719 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.716306 7007 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.001s
I20260808 01:50:57.716902 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:57.721709 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.722352 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:57.724507 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20260808 01:50:57.725230 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.729306 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:57.738260 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.740640 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:57.741804 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.741912 6896 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.746040 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.746140 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.747573 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.748855 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 218 to 219, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.755527 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.760898 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.765434 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.767854 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.769289 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.761528 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.775425 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.778707 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.786561 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.796080 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.802853 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.803236 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.804060 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20260808 01:50:57.805572 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.001s
I20260808 01:50:57.809854 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.812855 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.813714 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.813882 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.815352 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.815546 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.820669 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.823161 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20260808 01:50:57.826864 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 219: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.822089 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.829068 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:57.829224 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.829799 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.831897 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.832238 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.836148 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.836509 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.839949 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.843636 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 1 index: 225. (index mismatch)
I20260808 01:50:57.845081 7017 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:57.852794 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.863947 7017 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.866984 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.869211 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.873413 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20260808 01:50:57.876103 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:57.882570 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 224: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.882732 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.881218 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 2.224, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.887153 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.894973 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.898512 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.903116 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:57.907856 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 3.221, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:57.913689 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 219 to 220, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.927002 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.927388 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.929595 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20260808 01:50:57.932891 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:57.935194 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:57.939492 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:57.939838 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:57.941481 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20260808 01:50:57.942193 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 221. Preceding OpId from leader: term: 3 index: 222. (index mismatch)
I20260808 01:50:57.943977 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:57.941459 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20260808 01:50:57.942314 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 224. Preceding OpId from leader: term: 2 index: 225. (index mismatch)
I20260808 01:50:57.946547 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:57.946645 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.947906 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:57.949126 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:57.951967 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.954505 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.958003 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.964888 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.971686 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:57.974061 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 228: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:57.975975 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.977448 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:57.981524 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:57.994165 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 1.225, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.001358 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 1.222, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.007465 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 225. Preceding OpId from leader: term: 1 index: 226. (index mismatch)
I20260808 01:50:58.008265 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.010012 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20260808 01:50:58.012252 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.013069 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:58.014156 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:58.015676 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.016535 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.025792 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 1.220, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.028080 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.031930 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.037051 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.038004 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.030937 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.039964 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.041520 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.048228 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.051618 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.052309 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch)
I20260808 01:50:58.053709 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20260808 01:50:58.055253 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 225: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.060086 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.061736 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.068169 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.070047 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.082216 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.090595 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.096935 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.099874 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 220: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.103871 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20260808 01:50:58.105563 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 2.223, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.105871 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.001s
I20260808 01:50:58.107654 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.109092 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.112182 7017 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.119225 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 2.223, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.120332 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 3.222, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.122501 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 2.225, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.124549 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 222. Preceding OpId from leader: term: 3 index: 223. (index mismatch)
I20260808 01:50:58.126545 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.126971 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.127691 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:58.128010 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.130309 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.130657 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.137089 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 223. Preceding OpId from leader: term: 2 index: 224. (index mismatch)
I20260808 01:50:58.138556 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 229: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.138913 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.140663 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:58.142902 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.145095 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 223. Preceding OpId from leader: term: 2 index: 224. (index mismatch)
I20260808 01:50:58.147643 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.148554 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.148627 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 225. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20260808 01:50:58.110209 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.149662 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.150465 6896 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.149875 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 220 to 221, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.158742 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:58.160482 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.001s
I20260808 01:50:58.161964 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.167387 7005 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.170167 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.189163 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.190517 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.196754 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.197906 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 1.223, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.201651 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.205412 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20260808 01:50:58.206346 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.206707 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.207245 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.207379 7017 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:58.210985 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 1.226, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.220402 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 227. (index mismatch)
I20260808 01:50:58.221089 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.221485 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
I20260808 01:50:58.222278 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
I20260808 01:50:58.224090 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
I20260808 01:50:58.226815 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.226704 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.228286 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.231738 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
I20260808 01:50:58.247347 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 1 | 220 | Yes
B | A* B | 1 | 219 | Yes
Tablet 24ad2932d9a1489b96a63bc762f6a23f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
I20260808 01:50:58.251574 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.252986 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
master | A B* | | | Yes
A | A B* C~ | 2 | 220 | Yes
B | A B* C~ | 2 | 220 | Yes
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
I20260808 01:50:58.258816 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.259179 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 2.221, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.265491 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.266372 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 221. Preceding OpId from leader: term: 2 index: 222. (index mismatch)
I20260808 01:50:58.267994 6896 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20260808 01:50:58.267701 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.270764 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.271092 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.272670 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.273038 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:58.278441 7005 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 219 | Yes
B | A B* | 2 | 219 | Yes
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:50:58.280951 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 3.223, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
The consensus matrix is:
I20260808 01:50:58.283006 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.283959 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.284961 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.285349 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
I20260808 01:50:58.288298 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 223. Preceding OpId from leader: term: 3 index: 224. (index mismatch)
I20260808 01:50:58.288450 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.288919 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.289940 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
Config source | Replicas | Current term | Config index | Committed?
I20260808 01:50:58.294090 7017 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 1 | 223 | No
B | A* B | 1 | 222 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 6 | 0 | 0 | 4
I20260808 01:50:58.297104 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 4
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
I20260808 01:50:58.298586 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 24
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:50:58.300354 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 2.224, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.305142 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.313820 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.322906 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 226: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.323494 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.326406 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.328879 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 230: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.331856 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 224. Preceding OpId from leader: term: 2 index: 225. (index mismatch)
I20260808 01:50:58.332454 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.332850 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:50:58.332382 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.333128 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.331781 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.333274 6896 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.002s
I20260808 01:50:58.342324 7008 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.347931 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.350270 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.351989 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.354705 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.355010 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.353891 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.360296 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 221: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.367507 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.375358 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.376372 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 2.231, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.377116 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.381290 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20260808 01:50:58.381289 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 231. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20260808 01:50:58.379626 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 221 to 222, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.383812 7045 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:58.384371 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.385246 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:58.390036 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 1 index: 225. (index mismatch)
I20260808 01:50:58.392053 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:58.397152 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.401670 7017 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.401729 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.403586 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.407388 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 227: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.408665 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.412333 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 231: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.412415 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.424919 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.425886 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.427495 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 2.224, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.428427 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.431830 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20260808 01:50:58.431968 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 2.225, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.438879 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 225. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20260808 01:50:58.440064 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20260808 01:50:58.425886 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.442520 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 1.222, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:58.443615 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 227: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 227 but the committed config has opid_index of 228
I20260808 01:50:58.444906 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 224. Preceding OpId from leader: term: 2 index: 225. (index mismatch)
I20260808 01:50:58.446504 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.447813 6896 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:58.448859 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch)
I20260808 01:50:58.448740 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.450074 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 2 index: 227. (index mismatch)
I20260808 01:50:58.450208 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.448169 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.003s
I20260808 01:50:58.450042 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.451403 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.453135 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:58.452890 7008 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.001s
I20260808 01:50:58.457428 6896 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.458777 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.459110 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.466426 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.469532 7005 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.470633 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.470525 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.473410 7007 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.474025 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.479525 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 1.228, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.479655 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.484308 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 1.225, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.485687 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.487095 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 2.232, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.489161 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.490247 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.489988 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.496327 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.496752 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.502630 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 225. Preceding OpId from leader: term: 1 index: 226. (index mismatch)
I20260808 01:50:58.507040 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 232. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20260808 01:50:58.506814 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.505333 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.004s
I20260808 01:50:58.508873 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:58.511101 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 228. Preceding OpId from leader: term: 1 index: 229. (index mismatch)
I20260808 01:50:58.511085 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.511776 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.514346 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:58.514206 7017 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
W20260808 01:50:58.514689 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.515275 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:58.515010 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 3.224, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:50:58.515597 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:50:58.517335 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 222 but the committed config has opid_index of 223
I20260808 01:50:58.518204 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.518549 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.520080 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 222 to 223, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.522338 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 224. Preceding OpId from leader: term: 3 index: 225. (index mismatch)
I20260808 01:50:58.521878 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.525116 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.527012 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.527802 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.533838 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.533754 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:58.538635 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 228: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.539978 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 232: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.541370 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.542794 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.546509 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.548489 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 2.223, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.549247 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.547078 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 1.223, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.553107 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 2.225, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.559890 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.560251 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.561743 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.562067 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.562636 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.563503 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.565606 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 2 index: 227. (index mismatch)
I20260808 01:50:58.566207 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 223. Preceding OpId from leader: term: 2 index: 224. (index mismatch)
I20260808 01:50:58.571659 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.572274 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20260808 01:50:58.573175 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.573851 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:58.574631 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.576417 6896 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:58.576478 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20260808 01:50:58.584667 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 225. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20260808 01:50:58.583496 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.589167 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.592770 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.595343 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.596423 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.603180 6896 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:58.604403 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20260808 01:50:58.604729 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.605286 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.610021 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.610402 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.614830 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.617355 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.224: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.619421 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.621879 7005 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:58.624410 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.625415 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.626197 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 1.226, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.631624 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 227. (index mismatch)
I20260808 01:50:58.633939 6934 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:58.639838 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 223: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.640545 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.643386 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.646443 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.648355 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.650130 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.654659 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 3.225, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.656409 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.657196 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20260808 01:50:58.664146 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.664521 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.658761 6934 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:50:58.675099 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 225. Preceding OpId from leader: term: 3 index: 226. (index mismatch)
I20260808 01:50:58.677191 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.677444 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:58.651808 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.678709 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.682683 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.685261 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.686902 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.688199 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.689582 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.690796 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.692754 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 223 to 224, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.692710 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.696969 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.702812 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 233: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.710732 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.712013 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.714620 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.722358 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.728986 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.729351 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.729337 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20260808 01:50:58.731408 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 1.229, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.733923 6934 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:58.735356 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.740324 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 2 index: 227. (index mismatch)
I20260808 01:50:58.737300 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.741694 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:58.742321 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.745940 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 2.224, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.748670 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.750393 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.755327 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.755762 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.763067 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.764963 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 224. Preceding OpId from leader: term: 2 index: 225. (index mismatch)
I20260808 01:50:58.767588 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:58.772049 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.775316 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 1 index: 225. (index mismatch)
I20260808 01:50:58.777220 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20260808 01:50:58.782507 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.783694 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 229. Preceding OpId from leader: term: 1 index: 230. (index mismatch)
I20260808 01:50:58.784981 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.785178 6934 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:58.791061 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.793854 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.799925 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 229: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.802642 6896 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.786379 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.806742 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.807161 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.811960 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20260808 01:50:58.812153 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.819857 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:58.822271 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:58.821919 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.826468 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.830322 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.837234 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20260808 01:50:58.838883 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:50:58.838418 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.841107 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.841441 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.845561 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.847084 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.850116 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 3.226, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.829391 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20260808 01:50:58.852241 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.854262 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:58.854984 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.225: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.859205 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 226. Preceding OpId from leader: term: 3 index: 227. (index mismatch)
I20260808 01:50:58.858861 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.861788 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:58.863776 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.864121 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.862879 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20260808 01:50:58.857545 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.874531 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.876041 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.878593 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.879247 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.880715 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.890622 7007 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:58.892925 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.896185 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 234: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.898254 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.899329 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.899173 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.901117 7007 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.914429 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.915493 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:58.921130 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:58.934108 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.938815 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20260808 01:50:58.937281 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.941870 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:58.941151 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.942858 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.943817 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.944739 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 224 to 225, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 225 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.950611 7007 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.953572 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.961354 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:58.971258 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:58.973100 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 1.228, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.975936 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.230, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:58.976228 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:58.981081 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 228. Preceding OpId from leader: term: 1 index: 229. (index mismatch)
I20260808 01:50:58.981779 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:58.982169 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:58.982828 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:58.985442 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:58.986627 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 2 index: 236. (index mismatch)
I20260808 01:50:58.988269 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
W20260808 01:50:58.991833 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 228 but the committed config has opid_index of 229
I20260808 01:50:58.996775 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 1.225, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.001331 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 1 index: 231. (index mismatch)
I20260808 01:50:59.003130 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 225. Preceding OpId from leader: term: 1 index: 226. (index mismatch)
I20260808 01:50:59.004369 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:59.006716 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.009923 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.015393 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.017791 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:50:59.021126 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.024245 6934 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:50:59.025862 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.027539 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.006734 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.030881 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20260808 01:50:59.032099 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.036782 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.037284 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
W20260808 01:50:59.039196 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.042724 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.046057 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.047309 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20260808 01:50:59.050073 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.050458 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.047511 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.055560 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.057245 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.064883 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.066344 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 2.225, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.075834 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.076062 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.092469 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 230: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.102376 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 3.227, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.103152 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.105967 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.106715 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.112443 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.112579 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 235: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:50:59.112810 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.118479 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 227. Preceding OpId from leader: term: 3 index: 228. (index mismatch)
I20260808 01:50:59.119791 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 225. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20260808 01:50:59.122424 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.122776 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.122812 7007 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.001s
I20260808 01:50:59.123353 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20260808 01:50:59.130149 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.130856 6896 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.131397 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.131793 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.132875 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.134038 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.135406 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.136040 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.137282 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.226: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.155037 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.155704 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 225: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.156831 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 1.226, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.159746 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.160629 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.165252 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.172426 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20260808 01:50:59.174120 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.174492 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.177248 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.178273 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20260808 01:50:59.178736 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.178300 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
W20260808 01:50:59.179095 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.180012 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.181066 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.184643 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 227. (index mismatch)
I20260808 01:50:59.186163 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20260808 01:50:59.181782 6896 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.006s
I20260808 01:50:59.187743 7008 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:59.188742 7058 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.181434 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 225 to 226, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.191659 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.195262 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.195240 7007 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.195526 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.201365 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.201845 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.207783 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.208580 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.217792 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.219311 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 231: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.219223 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.231210 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.231060 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.236931 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 1.229, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.243199 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.244720 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.248302 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 2.226, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.250324 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.252939 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 3.228, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.259269 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 226. Preceding OpId from leader: term: 2 index: 227. (index mismatch)
I20260808 01:50:59.260035 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 228. Preceding OpId from leader: term: 3 index: 229. (index mismatch)
I20260808 01:50:59.260704 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.267450 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.267860 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.270916 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20260808 01:50:59.271545 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20260808 01:50:59.273165 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.001s
I20260808 01:50:59.274663 7007 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:59.278030 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.279346 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.279695 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.282294 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.285768 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.227: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.287055 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.268862 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.292182 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.293481 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.294550 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.295866 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.298440 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 229. Preceding OpId from leader: term: 1 index: 230. (index mismatch)
I20260808 01:50:59.299974 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.304821 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.307233 7058 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.309145 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.310906 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 236: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.321049 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.333845 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 226: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.338256 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.352133 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.358678 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.358790 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:50:59.359161 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.367405 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 226 to 227, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.368642 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.376211 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 2.231, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.376984 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.372834 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.382675 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
I20260808 01:50:59.383972 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.383769 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
W20260808 01:50:59.384366 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.384985 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.001s
I20260808 01:50:59.385607 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.391577 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20260808 01:50:59.392428 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 231. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20260808 01:50:59.392787 7058 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:59.391057 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.396090 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.396449 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.397038 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.398070 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20260808 01:50:59.398178 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.398147 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.402751 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
I20260808 01:50:59.402751 6896 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:59.404153 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:50:59.411486 6896 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.413733 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.414028 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.414857 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.422552 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.422254 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.423592 6896 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.424327 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.425812 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 232: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.431993 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.434768 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.434489 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:59.439203 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 230 but the committed config has opid_index of 231
I20260808 01:50:59.441262 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.230, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.445813 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 1 index: 231. (index mismatch)
I20260808 01:50:59.445546 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.447278 7058 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:50:59.449959 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.451436 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.454710 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 3.229, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.452899 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.456102 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.459684 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20260808 01:50:59.461824 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.462188 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.461982 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:50:59.468781 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.469126 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.470466 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 229. Preceding OpId from leader: term: 3 index: 230. (index mismatch)
I20260808 01:50:59.469774 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.471076 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.472378 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20260808 01:50:59.473487 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.473184 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.475857 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20260808 01:50:59.482254 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.486608 7007 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.480777 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.488848 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.490198 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.498662 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20260808 01:50:59.498258 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.500581 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.501212 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 237: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.502099 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:50:59.507629 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 1.228, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.508471 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.512513 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.512452 7008 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.516093 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 2.231, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.516300 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 2.232, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.518335 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 227: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.519078 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 233: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.517885 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.527151 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 227 to 228, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.530009 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 228. Preceding OpId from leader: term: 1 index: 229. (index mismatch)
I20260808 01:50:59.531708 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.532099 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.532781 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.532049 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.532662 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.539276 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 232. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20260808 01:50:59.540400 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 231. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20260808 01:50:59.541328 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:59.546779 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:59.547127 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.552039 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 2.231, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.552291 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.552902 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.551483 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.560075 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.565237 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.567384 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.567764 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.571022 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.574004 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.574100 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.574004 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.578486 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.578480 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 231. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20260808 01:50:59.581352 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 3.230, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.583323 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20260808 01:50:59.583912 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.585057 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.586014 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.589202 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.589540 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.588527 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20260808 01:50:59.591486 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
I20260808 01:50:59.592862 7058 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:50:59.593669 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20260808 01:50:59.593871 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:50:59.593151 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:59.594583 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.595582 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.596750 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.597563 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.599790 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:50:59.600579 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.605039 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.609699 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 230. Preceding OpId from leader: term: 3 index: 231. (index mismatch)
I20260808 01:50:59.611528 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.616452 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.619297 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20260808 01:50:59.620922 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:50:59.625236 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.626679 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.632054 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.633456 7007 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.637147 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.638032 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 1.229, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.619313 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.644421 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 234: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.651194 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.652088 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 238: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.654284 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.654646 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.654430 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.657800 7005 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.661295 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.667398 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.677160 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 229. Preceding OpId from leader: term: 1 index: 230. (index mismatch)
I20260808 01:50:59.678153 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.678946 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.682300 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.683422 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.687427 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.001s
I20260808 01:50:59.689905 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 228: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.695500 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.684278 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.703008 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.710318 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.724288 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.725155 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.726089 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.730476 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20260808 01:50:59.731973 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:50:59.733518 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.743139 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20260808 01:50:59.744684 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:59.748553 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.727933 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.754748 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.756474 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.761023 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.764389 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.767554 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.774852 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 235: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.794312 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:59.796980 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 2.232, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.798223 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 228 to 229, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.801196 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 232. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20260808 01:50:59.800756 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.801959 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.802976 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:59.803015 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.805380 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.809015 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.810880 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.818270 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.823254 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.826290 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20260808 01:50:59.828012 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:50:59.831821 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.834813 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.836867 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.851135 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.230, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.861492 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 2.239, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.862833 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.866104 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 239. Preceding OpId from leader: term: 2 index: 240. (index mismatch)
I20260808 01:50:59.866216 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 1 index: 231. (index mismatch)
I20260808 01:50:59.868431 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.868870 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 2.232, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:50:59.870891 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:50:59.874941 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.875948 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:50:59.878026 7005 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:50:59.879287 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 232. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20260808 01:50:59.881127 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:50:59.885782 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 3.231, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.885780 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 2.229, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.898530 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.898934 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.901751 6896 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.905606 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:50:59.906000 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:59.908610 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:50:59.909400 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 229. Preceding OpId from leader: term: 2 index: 230. (index mismatch)
I20260808 01:50:59.911188 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 231. Preceding OpId from leader: term: 3 index: 232. (index mismatch)
I20260808 01:50:59.911252 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.914851 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I20260808 01:50:59.916692 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:50:59.920897 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.919749 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.933072 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.938156 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:50:59.939772 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 51460.1us
I20260808 01:50:59.941387 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.942719 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.950354 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.962342 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.230: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:50:59.925451 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:50:59.970767 4770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 45306.4us
I20260808 01:50:59.971546 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 239: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.972169 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:50:59.972740 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.973378 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 229: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:50:59.915745 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:50:59.982426 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.983747 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.985029 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 229 to 230, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 230 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.986377 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.987735 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:50:59.994318 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:50:59.991264 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.006279 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20260808 01:51:00.012456 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:00.014624 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20260808 01:51:00.017524 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:51:00.022526 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.023416 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.031435 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.031886 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.032547 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.032239 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:00.032887 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.035053 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.036446 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 2.230, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.037503 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 3.232, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.040094 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.043478 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.045243 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.045588 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.049430 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:00.049603 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20260808 01:51:00.050806 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20260808 01:51:00.054144 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:51:00.055738 7007 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:00.058195 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.058535 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.060081 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.062199 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.062556 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.066591 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 232. Preceding OpId from leader: term: 3 index: 233. (index mismatch)
I20260808 01:51:00.067091 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.068673 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20260808 01:51:00.069829 6896 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:51:00.070303 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20260808 01:51:00.072968 6896 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:51:00.075387 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:51:00.076707 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.078987 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.080540 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 230. Preceding OpId from leader: term: 2 index: 231. (index mismatch)
I20260808 01:51:00.081830 7058 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.077243 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.094084 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 236: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.096911 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.102299 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
I20260808 01:51:00.102597 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.104118 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.107275 7007 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20260808 01:51:00.107318 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:00.108105 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.111099 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.118448 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.122373 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.122929 7005 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.127487 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.128575 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 240: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.131147 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.132737 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.137347 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.141367 7007 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.145268 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.146309 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.150534 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.154187 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.157168 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.160035 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.160300 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.165827 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.167739 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.170632 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.172139 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.174006 7005 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.176301 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.199844 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20260808 01:51:00.201241 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.231: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.209507 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.216513 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:51:00.217696 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.220288 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 230: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.231055 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 3.233, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.225370 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.236929 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.236158 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.242271 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.246727 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:00.249197 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.253137 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.254278 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.254736 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 233. Preceding OpId from leader: term: 3 index: 234. (index mismatch)
I20260808 01:51:00.255123 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20260808 01:51:00.261834 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:00.264323 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.001s
I20260808 01:51:00.260317 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 230 to 231, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 231 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.274458 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.278708 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.279255 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.283355 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.283514 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 2.231, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.288087 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20260808 01:51:00.289592 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:00.293229 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.293447 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.294175 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 2 index: 236. (index mismatch)
I20260808 01:51:00.297372 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20260808 01:51:00.299038 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:00.299261 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20260808 01:51:00.302759 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:00.305502 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.308609 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.309840 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:00.316224 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.321014 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.321384 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.323017 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20260808 01:51:00.308686 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.328774 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.330375 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.330679 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.331110 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.331727 7005 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.001s
I20260808 01:51:00.333794 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.340248 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.342732 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 231. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20260808 01:51:00.349807 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.351634 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 241: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.355095 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.360955 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 237: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.362043 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.362442 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.366003 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20260808 01:51:00.369639 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.370536 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.378799 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.381901 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.390635 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.403424 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.404794 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20260808 01:51:00.407681 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.410508 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.408799 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.414052 6896 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:51:00.416069 7007 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.428375 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.436065 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 231: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.442631 7005 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.442993 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.444394 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.445693 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.452441 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.456069 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.459401 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.232: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.467085 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.467475 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.473230 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.474828 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 2 index: 236. (index mismatch)
W20260808 01:51:00.473979 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.476142 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.476246 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.482452 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.484624 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.492468 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.495008 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20260808 01:51:00.495780 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.499377 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.503166 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.504566 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.497408 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:00.509943 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.510305 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.510702 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.511487 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20260808 01:51:00.512856 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.520727 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.520989 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
W20260808 01:51:00.521130 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.522578 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.522953 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.523875 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:00.526106 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 2 index: 236. (index mismatch)
I20260808 01:51:00.525943 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.527513 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:00.529016 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.531814 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.535545 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.539662 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.539815 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.547992 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.554253 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.555548 6896 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.559018 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20260808 01:51:00.562420 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:00.571048 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.573853 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.576366 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.589547 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.591189 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.592531 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 231 to 232, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.600370 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.609719 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.625520 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.626673 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.633919 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 242: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.645717 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 238: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.652256 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.661232 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.666260 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.668885 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 2.232, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.672367 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20260808 01:51:00.673914 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 232. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20260808 01:51:00.674048 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:00.675494 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20260808 01:51:00.680070 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 3.234, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.683851 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.690982 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20260808 01:51:00.691442 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.693442 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:00.692996 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:00.696208 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.233: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.702785 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 234. Preceding OpId from leader: term: 3 index: 235. (index mismatch)
I20260808 01:51:00.706149 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.708431 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:00.711669 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.719255 7005 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.719852 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.723393 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.727872 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.733448 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.239, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.738695 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20260808 01:51:00.739116 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.738870 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 2.243, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:00.739483 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.740229 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:00.747097 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.752506 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20260808 01:51:00.751827 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.757747 7007 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:00.760205 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.761951 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:00.764961 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 243. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20260808 01:51:00.766297 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20260808 01:51:00.766870 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:00.767907 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:00.771306 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20260808 01:51:00.772738 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.773283 7045 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:00.776510 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.783849 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.786324 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.799081 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.804893 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.804760 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.809056 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.817965 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.822914 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.824937 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 232: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.831979 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 239: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.832806 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.832100 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.838235 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.845556 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.832321 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.800771 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.856149 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.861811 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 243: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.864238 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.865124 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:00.877324 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.880426 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.895537 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.897433 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.901718 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.902094 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.902216 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:00.902951 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.903303 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.903268 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20260808 01:51:00.903370 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 232 to 233, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.904953 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:00.907349 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20260808 01:51:00.908857 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:00.912966 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.923454 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.925877 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.932884 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.934857 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.942404 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 240: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.946096 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.948081 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.956635 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.964705 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 3.235, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.968990 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20260808 01:51:00.970504 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.973546 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.974169 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:00.977830 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.980438 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20260808 01:51:00.982327 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.982676 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.983109 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:00.987071 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:00.987406 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 235. Preceding OpId from leader: term: 3 index: 236. (index mismatch)
W20260808 01:51:00.987448 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.988765 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20260808 01:51:00.987368 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:00.990725 7007 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:00.996363 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.996716 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.997248 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:00.999357 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:00.999989 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:00.999435 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.001675 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.002764 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.003512 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.009907 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.011062 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.012247 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:00.998524 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.018221 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.023900 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.024001 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.027699 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.032696 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20260808 01:51:01.034263 7007 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:01.035068 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.035506 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.040442 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.058267 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 244: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.060391 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.066148 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20260808 01:51:01.068109 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.069801 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.067767 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.006s
I20260808 01:51:01.076553 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.078007 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.079620 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.083622 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.086473 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.086653 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
W20260808 01:51:01.086925 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.086239 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.088258 7007 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20260808 01:51:01.093914 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.094683 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 241: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.094507 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.103376 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.104944 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 233: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.081151 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.126773 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.131721 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
I20260808 01:51:01.133988 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.136456 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.137749 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.142022 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.145103 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20260808 01:51:01.146550 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:01.146665 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 3.236, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.147778 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20260808 01:51:01.149518 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20260808 01:51:01.150194 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.151197 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:01.154964 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.158113 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 236. Preceding OpId from leader: term: 3 index: 237. (index mismatch)
I20260808 01:51:01.154978 7005 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.158278 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.159544 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.163147 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.166685 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.176853 7008 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:01.177678 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.184268 7008 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.185108 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.187508 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.191505 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.195067 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.196798 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.202097 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.176760 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.204097 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.205520 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 233 to 234, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.206843 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.210525 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 245: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.218001 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
I20260808 01:51:01.219532 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.221942 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.224916 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.225111 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.227093 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.230365 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.236029 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 243. (index mismatch)
I20260808 01:51:01.237506 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:01.245669 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.246079 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.246235 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.254736 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.256502 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.260083 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 242: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.261549 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.273090 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.274996 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20260808 01:51:01.276713 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 2.239, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:01.281961 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.284323 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.282738 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:01.283483 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.239, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.290102 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 3.237, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.292440 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.292266 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.292296 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.299422 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
W20260808 01:51:01.299870 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.300868 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:01.303902 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.305971 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.310294 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 237. Preceding OpId from leader: term: 3 index: 238. (index mismatch)
I20260808 01:51:01.309590 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.311655 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 239. Preceding OpId from leader: term: 2 index: 240. (index mismatch)
I20260808 01:51:01.312964 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
Master Summary
I20260808 01:51:01.316277 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.316469 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
W20260808 01:51:01.316636 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.317830 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:01.318537 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.318956 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.319761 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.320220 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
I20260808 01:51:01.330229 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:51:01.318634 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.326007 7045 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
I20260808 01:51:01.337667 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 230 | Yes
B | A B* | 2 | 229 | Yes
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:01.342971 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 1 index: 244. (index mismatch)
I20260808 01:51:01.344175 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:01.344633 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:01.345870 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.345806 7005 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:01.347494 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20260808 01:51:01.349592 7005 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20260808 01:51:01.351262 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 1 | 234 | Yes
B | A* B | 1 | 233 | Yes
C | [config not available] | | |
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:01.354276 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 3 | 232 | Yes
B | A B* | 3 | 231 | Yes
Tablet 24ad2932d9a1489b96a63bc762f6a23f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
I20260808 01:51:01.334100 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 234 | No
B | A B* | 2 | 233 | No
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 241 | No
B | A* B | 2 | 240 | No
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 1 | 232 | No
B | A B* | 1 | 231 | No
C | [config not available] | | |
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 234 | No
B | A B* | 2 | 233 | Yes
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 1 | 237 | Yes
B | A* B | 1 | 236 | Yes
C | [config not available] | | |
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:01.363874 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 235 | No
B | A B* C~ | 2 | 234 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 1 | 0 | 0 | 9
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 6
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 26
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:01.370909 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.385113 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.389194 7004 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.390159 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.392390 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.395046 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.401908 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.402709 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 243: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.406001 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:01.405601 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.411639 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.413139 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 246: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.418969 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.425565 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.422544 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.424522 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.445410 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.447645 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.423542 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.450258 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.458338 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 2.247, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.463342 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 247. Preceding OpId from leader: term: 2 index: 248. (index mismatch)
I20260808 01:51:01.465039 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:01.460278 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.470067 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.472146 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.480304 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.484764 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.486030 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.486902 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.487627 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.488495 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 2.239, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.489867 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 247: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.491815 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
I20260808 01:51:01.494349 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 239. Preceding OpId from leader: term: 2 index: 240. (index mismatch)
I20260808 01:51:01.494572 7080 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:01.495051 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.495445 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.496198 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:01.503134 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.502995 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.504860 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.507433 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20260808 01:51:01.508733 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:01.508740 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.514534 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.514744 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.515547 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.517019 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.516304 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.522642 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.524956 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.524667 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.532694 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.526396 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.533687 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.533931 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
W20260808 01:51:01.534102 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.535318 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:01.535494 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 234 to 235, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.541148 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.542894 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.550884 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 244: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.555256 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 3.238, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.555169 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.557744 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 2.235, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.559464 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20260808 01:51:01.560299 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 238. Preceding OpId from leader: term: 3 index: 239. (index mismatch)
I20260808 01:51:01.560932 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.561667 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.561957 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:01.562350 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.567471 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.567895 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.569726 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 2.239, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.570425 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.572989 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.574785 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 235. Preceding OpId from leader: term: 2 index: 236. (index mismatch)
I20260808 01:51:01.577294 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.578099 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20260808 01:51:01.579123 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 239. Preceding OpId from leader: term: 2 index: 240. (index mismatch)
I20260808 01:51:01.579442 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.581148 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
W20260808 01:51:01.577785 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.582319 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.586465 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.588228 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.588421 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.597579 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.598726 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.598997 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.236: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.599989 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.601889 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.602120 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.605254 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 235: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.605880 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.610236 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20260808 01:51:01.611127 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.611563 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.611605 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:01.613240 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.619062 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.621163 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20260808 01:51:01.622709 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:01.624097 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.625175 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:01.621922 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.626012 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:01.626956 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.630263 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.632757 7004 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.635006 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.634857 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.638935 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.641580 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
I20260808 01:51:01.642949 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.643306 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.643402 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20260808 01:51:01.644412 7005 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:01.646205 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:01.648252 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.649917 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 235 to 236, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.651117 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.654217 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 248: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.656049 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.664152 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.666451 7007 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.668897 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.670979 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.678275 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.680531 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.681416 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.682044 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.239, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.688020 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.688398 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.688462 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.690066 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20260808 01:51:01.667363 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20260808 01:51:01.692055 7007 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.001s
I20260808 01:51:01.692365 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.693544 7004 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:01.695099 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.699337 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 2.236, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.701035 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 3.239, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.704948 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 236. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20260808 01:51:01.705390 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.707540 7080 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20260808 01:51:01.709640 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
I20260808 01:51:01.712170 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 239. Preceding OpId from leader: term: 3 index: 240. (index mismatch)
I20260808 01:51:01.712195 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 243. (index mismatch)
I20260808 01:51:01.712581 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.713670 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:01.714123 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.714483 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.716383 7005 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:01.717674 7005 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:01.722271 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20260808 01:51:01.722564 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.725406 7007 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.727631 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.731063 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:01.731195 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.735116 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 245: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.738438 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.738844 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.739836 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.741146 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.745064 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.747826 7008 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.758033 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.759083 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.759799 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 236: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.765074 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.768208 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.769304 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 249: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.770205 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.773234 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.785503 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.785969 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.785383 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.237: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.780634 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.787613 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.791488 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.796928 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.797820 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.799130 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.800696 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 236 to 237, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.802346 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20260808 01:51:01.802110 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:01.804803 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20260808 01:51:01.807457 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:01.812955 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.817471 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:01.820714 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.821534 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.824280 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.825896 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 3.240, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.834686 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.835264 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.843542 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.844079 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.846489 7080 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.848780 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20260808 01:51:01.849299 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.850484 7045 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:01.857295 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.858004 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.857985 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.859721 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.863173 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.864449 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.865868 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.867882 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 246: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.870368 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.875156 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20260808 01:51:01.876339 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 1 index: 244. (index mismatch)
I20260808 01:51:01.876398 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.881119 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 2.250, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:01.881294 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:01.884063 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.885468 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.888361 7004 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.888577 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.891031 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.896446 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.897449 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:01.897861 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:01.900573 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.905784 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.907593 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 250. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20260808 01:51:01.909024 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:01.915607 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.917769 7007 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:01.922519 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.927369 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.927949 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.928668 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:01.933691 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 250: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.933769 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20260808 01:51:01.936923 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:01.939000 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 240. Preceding OpId from leader: term: 3 index: 241. (index mismatch)
I20260808 01:51:01.940819 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:01.946411 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:01.946628 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.958374 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:01.959887 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 247: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:01.958444 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.961053 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20260808 01:51:01.963093 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.970904 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20260808 01:51:01.972268 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:01.974488 7007 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.986788 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20260808 01:51:01.988870 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:01.998298 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.002558 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.007184 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 237: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.012058 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:02.017318 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 2.243, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.018673 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.023696 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 2.251, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.026557 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.031596 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20260808 01:51:02.011370 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.033461 7004 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.034011 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 251. Preceding OpId from leader: term: 2 index: 252. (index mismatch)
I20260808 01:51:02.035336 7004 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:02.044593 7045 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.046280 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.049659 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.050042 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.050647 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 243. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20260808 01:51:02.050712 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.053344 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20260808 01:51:02.055328 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 251: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.055814 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.057077 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.001s
I20260808 01:51:02.060389 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.060457 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:02.062047 7080 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.076594 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.083228 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.091837 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.093833 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.104972 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.107468 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.110273 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.114331 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.115355 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.116585 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.117939 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.118983 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 237 to 238, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.120054 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.132629 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.133764 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.135006 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.135941 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.136350 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.138047 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.143002 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20260808 01:51:02.144200 7045 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:02.145519 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.145910 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.151244 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.152587 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.161582 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 248: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.163949 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 2.252, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.166924 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.167202 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.170590 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.171007 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 2.243, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.171567 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.172019 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 3.241, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.177841 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.180109 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20260808 01:51:02.181872 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.182243 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.182337 7080 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:02.183286 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 241. Preceding OpId from leader: term: 3 index: 242. (index mismatch)
I20260808 01:51:02.185220 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:02.185516 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 243. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20260808 01:51:02.186491 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.186869 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.187463 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.187777 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.188330 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:02.188354 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
W20260808 01:51:02.188665 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.188524 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:02.191035 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 252. Preceding OpId from leader: term: 2 index: 253. (index mismatch)
I20260808 01:51:02.192358 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:02.192209 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.195883 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
I20260808 01:51:02.197737 7007 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.197422 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20260808 01:51:02.199549 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20260808 01:51:02.200217 7080 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.202426 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.204694 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.214599 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.216787 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.217216 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.219336 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.221206 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.222133 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:02.224082 7008 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.231382 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.231878 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.247181 7005 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.257923 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.258906 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.262609 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 252: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.262228 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.269085 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.269610 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 1.249, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.270737 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.273216 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20260808 01:51:02.276361 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.281973 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.283694 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.285686 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20260808 01:51:02.289887 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.292384 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.297190 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:02.297583 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:02.288328 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.304291 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.307072 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.310226 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.315644 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.300952 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.322472 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.323964 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.318801 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.327524 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 249: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.329147 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 238 to 239, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.341764 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.343608 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 2.253, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.345363 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.351509 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 3.242, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.347672 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20260808 01:51:02.353979 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:02.356703 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.357884 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20260808 01:51:02.358203 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.359630 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.359543 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.361205 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 242. Preceding OpId from leader: term: 3 index: 243. (index mismatch)
I20260808 01:51:02.360996 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.362949 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.367931 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 243. (index mismatch)
I20260808 01:51:02.368815 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:02.370488 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20260808 01:51:02.374603 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 253: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.372861 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.382400 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.383123 7080 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.386865 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.390096 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.395684 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:02.395678 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
W20260808 01:51:02.396075 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.397279 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:02.397040 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.402112 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.403957 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.410645 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.413533 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 250: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.415395 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.416846 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.418112 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.425730 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20260808 01:51:02.427220 7045 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:02.431066 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.434495 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.435998 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.437260 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.439540 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.440724 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.454519 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.460294 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.461839 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.465634 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 2.239, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.468967 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20260808 01:51:02.471509 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.471875 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.472506 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:02.472832 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20260808 01:51:02.474216 7045 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:02.474274 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 2.243, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.480985 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.481863 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.483625 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.483722 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.483989 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.485442 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.486543 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.487208 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 243. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20260808 01:51:02.488759 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:02.489290 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 239. Preceding OpId from leader: term: 2 index: 240. (index mismatch)
I20260808 01:51:02.490773 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 251: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.493850 7080 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20260808 01:51:02.495601 7080 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.496780 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.497344 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.498158 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.498557 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.499609 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.502313 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.507040 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20260808 01:51:02.507910 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.508291 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.509282 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.001s
I20260808 01:51:02.509671 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.519263 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.529026 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.533490 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.537607 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.539194 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.547111 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.553006 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 239: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.553258 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 1.243, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.559755 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.560187 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.556886 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.575901 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 3.243, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.578869 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20260808 01:51:02.580515 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.581427 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.586465 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 243. Preceding OpId from leader: term: 1 index: 244. (index mismatch)
I20260808 01:51:02.586632 5070 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20260808 01:51:02.589581 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 243. Preceding OpId from leader: term: 3 index: 244. (index mismatch)
I20260808 01:51:02.591933 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.001s
I20260808 01:51:02.595088 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:02.595708 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:02.596686 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:02.600090 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.602335 7045 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.605783 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 254: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.607677 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:02.608088 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.591665 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.240: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:02.609577 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.614629 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:02.611999 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.614158 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:02.615406 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.601243 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.617120 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.618331 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.620786 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.622085 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.624085 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.624639 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.638689 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.644309 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.644269 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.649364 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 239 to 240, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.651500 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.662693 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.664685 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 2.255, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.668179 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 2.240, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.670440 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.670862 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.672724 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 255. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20260808 01:51:02.668179 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.672714 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20260808 01:51:02.677145 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.678973 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:02.680271 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:02.680485 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.682130 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20260808 01:51:02.682107 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.683470 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.683291 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.684666 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.688009 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
I20260808 01:51:02.690552 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:02.693380 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.696159 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.699847 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.704710 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.708338 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 240. Preceding OpId from leader: term: 2 index: 241. (index mismatch)
I20260808 01:51:02.708317 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.709760 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I20260808 01:51:02.710924 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20260808 01:51:02.712312 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:02.719986 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.718108 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.723582 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.699663 7080 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.725268 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.722506 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.727662 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.728222 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.731309 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.241: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.735623 5070 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20260808 01:51:02.735999 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 255: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.737847 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.740103 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 252: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.741905 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.742206 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.745594 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.755287 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.760283 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 3.244, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.762315 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.762804 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 2.256, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.767889 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.771184 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.776784 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.777886 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 244. Preceding OpId from leader: term: 3 index: 245. (index mismatch)
I20260808 01:51:02.779371 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:02.781522 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.781920 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.782253 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.791832 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.794206 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.801810 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 240: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.806721 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
I20260808 01:51:02.810561 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:02.812611 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.813409 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.815899 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.816708 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.816979 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.818141 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.818379 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 240 to 241, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.819360 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.826097 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.814945 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20260808 01:51:02.830559 7045 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:02.832978 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 2.247, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.839799 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:02.839335 7045 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:02.840165 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.849512 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.853322 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 1.249, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.854205 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.855245 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 247. Preceding OpId from leader: term: 2 index: 248. (index mismatch)
I20260808 01:51:02.855933 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:02.856696 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:02.862636 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.863056 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.867009 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20260808 01:51:02.871304 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 256: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.874229 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.868392 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.010s
I20260808 01:51:02.881523 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.886375 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.891773 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.892180 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.894222 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20260808 01:51:02.864920 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.894547 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.897217 7005 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:02.901779 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.903159 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:02.903648 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 2.241, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.908332 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.912503 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.912884 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.917212 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 241. Preceding OpId from leader: term: 2 index: 242. (index mismatch)
I20260808 01:51:02.916996 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.918799 7005 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I20260808 01:51:02.920538 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.923872 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20260808 01:51:02.925352 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:02.925487 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.925184 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:02.934971 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.931667 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.936237 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 3.245, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.937433 6971 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.941399 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 245. Preceding OpId from leader: term: 3 index: 246. (index mismatch)
I20260808 01:51:02.942219 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.942603 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.943280 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:02.945327 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.947721 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.952114 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.954066 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20260808 01:51:02.954543 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 253: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:02.956175 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:02.966264 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20260808 01:51:02.965490 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.242: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.973855 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.975869 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.976737 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 2.247, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:02.978482 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 241: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.984637 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.985443 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:02.987830 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:02.988526 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:02.979354 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.010s
I20260808 01:51:02.991272 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:02.994779 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:02.995754 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.000528 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 247. Preceding OpId from leader: term: 2 index: 248. (index mismatch)
I20260808 01:51:03.003963 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.005548 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.009856 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20260808 01:51:03.011209 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:03.011304 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 241 to 242, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.018169 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:03.017824 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.019799 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 257: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.021149 7080 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.027415 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:03.027777 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.031287 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.037588 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.042898 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.044445 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.058487 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.059481 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.067298 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.071920 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20260808 01:51:03.073385 7080 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:03.074805 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.076025 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.082201 7008 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.087327 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.091100 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.092797 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.097559 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20260808 01:51:03.105829 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.001s
I20260808 01:51:03.110121 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.112401 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.113955 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.115526 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.118203 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.124616 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.115329 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.133980 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20260808 01:51:03.138537 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.001s
I20260808 01:51:03.141341 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.143554 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.146342 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.149742 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.150112 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.153795 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.154153 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.154752 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:03.154760 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 255. (index mismatch)
W20260808 01:51:03.155059 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.156327 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:03.158390 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.162177 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20260808 01:51:03.166846 7045 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:03.168264 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.174558 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20260808 01:51:03.174335 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.176738 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:03.179378 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.180459 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.186725 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.185155 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.188068 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.185155 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.182627 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 3.246, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.200562 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.208045 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
I20260808 01:51:03.208287 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20260808 01:51:03.209996 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 246. Preceding OpId from leader: term: 3 index: 247. (index mismatch)
I20260808 01:51:03.176049 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.212275 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:03.213094 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:03.213554 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:03.222311 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.222536 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 242 to 243, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.225229 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.226366 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 258: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.230342 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.230959 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.245098 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 254: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.261355 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.261109 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.263368 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 2.243, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.261109 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.274333 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.286943 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:03.294029 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.294484 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:03.297744 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 72326.6us
I20260808 01:51:03.301473 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:03.301076 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.301998 7008 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.303488 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.306782 7005 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.309098 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20260808 01:51:03.310673 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:03.313294 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20260808 01:51:03.313150 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.314992 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:03.316457 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.318095 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.319859 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.321851 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 243. Preceding OpId from leader: term: 2 index: 244. (index mismatch)
I20260808 01:51:03.324451 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I20260808 01:51:03.323731 7045 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.324092 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.325847 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:03.301849 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.333391 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.334354 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.334710 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.336109 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.336591 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:03.348315 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.356577 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 255: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.358429 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20260808 01:51:03.358260 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.244: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.361328 7080 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.001s
I20260808 01:51:03.364632 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.366658 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.374573 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20260808 01:51:03.375110 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.376170 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:03.377331 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20260808 01:51:03.379845 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:03.383387 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.388478 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.395392 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:03.395709 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.396134 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.387072 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.397907 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.400300 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.413008 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.414250 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:03.415239 4770 catalog_manager.cc:5220] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 243: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 243 but the committed config has opid_index of 244
I20260808 01:51:03.420651 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.425607 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.427846 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:03.432626 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 259: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.434679 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
W20260808 01:51:03.441790 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.443421 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:03.452672 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.454362 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.455811 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.460309 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 243 to 244, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.462054 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 256: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.463052 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.466876 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.462338 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.469493 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.470769 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.483031 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.507261 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.520244 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.524917 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.529043 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.529517 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.532347 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20260808 01:51:03.533818 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:03.537948 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 2 index: 245. (index mismatch)
I20260808 01:51:03.539323 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20260808 01:51:03.539510 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.541147 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.536468 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 2.250, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.545069 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.546823 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.245: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.553931 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 250. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20260808 01:51:03.556164 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.557056 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:03.561492 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.561854 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.564939 7080 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.566205 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.566923 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 244: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.567255 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.571676 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20260808 01:51:03.572782 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.573745 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.574420 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.578383 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:03.587445 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.591241 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 3.247, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.596416 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.608448 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.610322 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.610702 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.613590 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 247. Preceding OpId from leader: term: 3 index: 248. (index mismatch)
I20260808 01:51:03.615207 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:03.632515 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.642941 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.643087 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.644869 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.647274 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.654512 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.658427 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 244 to 245, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.663103 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20260808 01:51:03.658921 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.664208 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.667572 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:03.665407 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.676385 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.686849 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.701422 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20260808 01:51:03.702842 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.703250 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.704767 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.001s
I20260808 01:51:03.711414 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.715771 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.742938 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 2.245, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.748900 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.749289 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.751245 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 245. Preceding OpId from leader: term: 2 index: 246. (index mismatch)
I20260808 01:51:03.752463 7080 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20260808 01:51:03.758754 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.760349 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.246: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.761801 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.766445 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.771736 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20260808 01:51:03.773120 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:03.773360 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 257: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.774955 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 260: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.780403 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.783357 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.791281 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 245: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.792397 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.796443 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 255. (index mismatch)
I20260808 01:51:03.797888 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:03.804557 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.805280 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.813436 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 2.247, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.814982 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.817394 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.823257 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 2.250, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.829473 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 247. Preceding OpId from leader: term: 2 index: 248. (index mismatch)
I20260808 01:51:03.781705 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.827087 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.833320 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.833921 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:03.837240 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 250. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20260808 01:51:03.845785 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:03.847502 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.850018 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.853502 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.860800 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:03.870292 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.878729 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.883484 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.887936 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20260808 01:51:03.889298 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:03.896422 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.898252 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 245 to 246, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:03.899799 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.899883 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.900866 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.906000 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.914526 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 261: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.930379 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:03.932801 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.934191 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.940312 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 2.246, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:03.946389 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 246. Preceding OpId from leader: term: 2 index: 247. (index mismatch)
I20260808 01:51:03.947959 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20260808 01:51:03.949263 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 2.262, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.955752 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 1.249, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.958595 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 262. Preceding OpId from leader: term: 2 index: 263. (index mismatch)
I20260808 01:51:03.958510 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:03.959390 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 2.251, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:03.960721 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.961099 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.962530 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:03.965978 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:03.956717 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.247: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:03.966336 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.969390 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 251. Preceding OpId from leader: term: 2 index: 252. (index mismatch)
I20260808 01:51:03.971092 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:03.972239 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 249. Preceding OpId from leader: term: 1 index: 250. (index mismatch)
I20260808 01:51:03.974524 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:03.974869 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:03.976450 7080 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:03.979008 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.976939 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.985131 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.982254 7008 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:03.995800 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:03.998631 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.004680 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 3.248, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.006426 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.009512 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.011493 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 248. Preceding OpId from leader: term: 3 index: 249. (index mismatch)
I20260808 01:51:04.011291 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.012864 7008 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:04.015450 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 262: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.021696 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.022115 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.033018 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.038053 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.038543 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.039516 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 246 to 247, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.041215 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.043563 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.048450 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.049804 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20260808 01:51:04.049932 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.051415 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:04.066476 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.070056 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.080715 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.089493 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 258: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.101842 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.102270 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.102072 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20260808 01:51:04.106065 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:04.106770 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.114506 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.116487 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.132226 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.136041 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.141837 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 2.252, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.146687 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20260808 01:51:04.149106 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 252. Preceding OpId from leader: term: 2 index: 253. (index mismatch)
I20260808 01:51:04.149788 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:04.151952 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.154233 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:04.156966 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 2.251, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.158016 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.135947 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 2.247, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.161962 5070 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20260808 01:51:04.161283 7080 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.162211 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
I20260808 01:51:04.163486 7008 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:04.163867 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:04.171411 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.175345 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.175771 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.178045 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.179620 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 247. Preceding OpId from leader: term: 2 index: 248. (index mismatch)
I20260808 01:51:04.181197 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 251. Preceding OpId from leader: term: 2 index: 252. (index mismatch)
I20260808 01:51:04.181756 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.182992 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:04.184147 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.185348 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.185969 7008 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.191810 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20260808 01:51:04.193147 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.194576 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 263: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.200958 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.202741 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.204002 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.214670 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.215562 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.217337 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.219389 7117 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:04.222141 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.224996 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.231046 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.234200 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.235122 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.235265 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.234436 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.247645 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.247645 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.248: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.259505 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 1 index: 260. (index mismatch)
I20260808 01:51:04.260960 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:04.268924 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 3.249, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.271837 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 2.253, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.277227 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 247: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.278847 5070 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 249. Preceding OpId from leader: term: 3 index: 250. (index mismatch)
I20260808 01:51:04.280050 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.281134 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:04.285465 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.285719 7008 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:04.290751 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.297897 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.298348 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
W20260808 01:51:04.299225 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.299901 7117 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:04.301012 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.304884 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20260808 01:51:04.307726 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:04.312320 7117 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.314808 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 247 to 248, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.316630 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.320209 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 259: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.318315 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.323577 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.323256 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.330204 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.340440 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.340333 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.343860 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.344242 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.346333 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20260808 01:51:04.348074 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:04.337508 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
I20260808 01:51:04.349689 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:51:04.355345 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 1 | 253 | Yes
B | A* B C~ | 1 | 252 | Yes
C | [config not available] | | |
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 3 | 247 | Yes
B | A B* | 3 | 247 | Yes
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:04.367511 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 259 | Yes
B | A* B | 2 | 260 | Yes
C | [config not available] | | |
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 247 | Yes
B | A B* | 2 | 247 | Yes
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
I20260808 01:51:04.381819 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B | 1 | 256 | Yes
I20260808 01:51:04.384397 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
B | A* B C~ | 1 | 257 | Yes
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:04.386029 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 249 | Yes
B | A B* | 2 | 249 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 4 | 0 | 0 | 6
I20260808 01:51:04.387465 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 10
Third Quartile | 10
Maximum | 10
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 29
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:04.397609 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:04.399593 4770 catalog_manager.cc:5220] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 253 but the committed config has opid_index of 254
I20260808 01:51:04.401173 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.406090 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 2.248, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.406281 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:04.413908 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 2.252, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.411176 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.432569 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20260808 01:51:04.432967 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.434057 5070 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
W20260808 01:51:04.433503 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.435359 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 252. Preceding OpId from leader: term: 2 index: 253. (index mismatch)
I20260808 01:51:04.436081 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 264: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.419255 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.438805 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.001s
I20260808 01:51:04.440621 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20260808 01:51:04.441759 7117 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.001s
I20260808 01:51:04.447906 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 3.250, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.450241 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.450742 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.450444 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 1.260, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.453326 5070 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.453930 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.459800 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20260808 01:51:04.461007 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 250. Preceding OpId from leader: term: 3 index: 251. (index mismatch)
I20260808 01:51:04.462551 7008 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.001s
I20260808 01:51:04.464429 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.464766 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.464111 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.467903 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 260. Preceding OpId from leader: term: 1 index: 261. (index mismatch)
I20260808 01:51:04.468365 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.468760 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.472074 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:04.473830 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.475610 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:04.477988 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.479147 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20260808 01:51:04.480571 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:04.475775 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 2.265, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.482481 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.483614 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.490020 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.491212 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.492535 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.493436 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.494763 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 265. Preceding OpId from leader: term: 2 index: 266. (index mismatch)
I20260808 01:51:04.498459 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:04.500795 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.501029 5070 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.501502 7117 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.506644 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20260808 01:51:04.507289 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 260: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.509311 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.512545 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.497905 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.516179 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.513204 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.517359 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.001s
I20260808 01:51:04.520032 7008 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.521034 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.530369 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.531879 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.531749 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.535597 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.249: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.538728 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.539682 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.541303 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.550150 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 2.253, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.550766 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.556988 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.557369 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.568325 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 265: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.571655 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.573331 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20260808 01:51:04.573354 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.574645 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.575613 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:04.575795 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 248 to 249, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.576778 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.579577 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.584057 7008 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.585691 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.594805 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.598920 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.601183 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.602331 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 1.261, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.605829 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20260808 01:51:04.610918 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:04.611270 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 261. Preceding OpId from leader: term: 1 index: 262. (index mismatch)
I20260808 01:51:04.613164 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:04.619966 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.620712 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.621675 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.622530 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.626579 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20260808 01:51:04.622918 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.626943 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.627318 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.628139 7080 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20260808 01:51:04.631134 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 2.255, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.633862 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 2.250, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.634140 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.637166 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 3.251, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.641742 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 261: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.642597 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20260808 01:51:04.648702 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.648078 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:04.649072 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.649000 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 2.266, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.649876 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 251. Preceding OpId from leader: term: 3 index: 252. (index mismatch)
I20260808 01:51:04.649662 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.650839 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:04.651955 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.653133 7117 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:04.654438 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 250. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20260808 01:51:04.653960 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.665187 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 266. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20260808 01:51:04.666496 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.666894 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.667724 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:04.668646 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 255. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20260808 01:51:04.669492 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.669890 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.676396 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.678566 7117 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:04.680543 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 1.262, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.681274 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.684228 7008 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.686411 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:04.690934 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.690855 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.691953 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 266: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.696319 5070 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.699668 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.700057 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.704126 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20260808 01:51:04.705823 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.001s
I20260808 01:51:04.711477 7117 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.715222 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.715893 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 249: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.719457 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.713652 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.725217 5070 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.725909 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.731330 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.734711 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 262: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.736510 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.738628 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.741582 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.742304 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.742446 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.733145 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 249 to 250, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.749305 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.757364 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.775876 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.777598 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.778779 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.781044 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.782311 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.784126 5070 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.791178 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.798384 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.802250 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 2.250, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.803681 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20260808 01:51:04.805507 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:04.807745 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 250. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20260808 01:51:04.809096 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20260808 01:51:04.810678 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.811030 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.811592 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20260808 01:51:04.812991 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:04.813257 7120 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.817224 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.819568 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.251: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.822324 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 1 index: 260. (index mismatch)
I20260808 01:51:04.822398 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.827363 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.830116 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:04.835177 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.835830 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 250: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.838398 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 2.256, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.839217 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.842976 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 3.252, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.846354 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 2.251, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.848295 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 252. Preceding OpId from leader: term: 3 index: 253. (index mismatch)
I20260808 01:51:04.851621 7117 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:04.853140 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.854230 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20260808 01:51:04.855366 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.856901 5070 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
W20260808 01:51:04.857415 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.856762 7117 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:04.859947 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.861464 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.861505 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 250 to 251, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.854305 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 251. Preceding OpId from leader: term: 2 index: 252. (index mismatch)
I20260808 01:51:04.863090 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.865368 7008 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:04.867256 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:04.867496 7008 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.870297 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.871397 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.877879 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.879608 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.883152 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.884752 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.885007 7117 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.887578 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.895538 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.884775 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20260808 01:51:04.895644 5070 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.898545 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:04.899946 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.903573 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.904204 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.910862 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 267: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.915771 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.918057 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.928422 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:04.929255 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 2.251, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.936348 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.936746 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.938311 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 2.255, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.938696 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 251. Preceding OpId from leader: term: 2 index: 252. (index mismatch)
I20260808 01:51:04.940092 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.942994 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 263: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:04.943753 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 1.260, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:04.941895 7120 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I20260808 01:51:04.947271 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.948638 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.949920 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.952524 7120 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.957406 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:04.956964 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:04.957804 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.958796 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 260. Preceding OpId from leader: term: 1 index: 261. (index mismatch)
I20260808 01:51:04.962010 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 255. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20260808 01:51:04.960239 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.002s
I20260808 01:51:04.964581 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:04.966558 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 251: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.967658 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:04.976029 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 2.268, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:04.978294 5070 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.252: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:04.980248 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.983302 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:04.983666 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:04.985558 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.987212 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:04.998787 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20260808 01:51:05.001144 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.016824 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.019129 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.020215 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 251 to 252, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.022508 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.023237 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.026338 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.027814 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.039775 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.040278 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.042661 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 2.252, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.051455 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20260808 01:51:05.053212 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 255. (index mismatch)
I20260808 01:51:05.054438 5070 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 1.264, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.056102 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 3.253, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.054804 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 252. Preceding OpId from leader: term: 2 index: 253. (index mismatch)
I20260808 01:51:05.059018 7120 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:05.059373 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:05.060715 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:05.060663 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
W20260808 01:51:05.061048 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.061261 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20260808 01:51:05.062772 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:05.062805 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20260808 01:51:05.064170 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.069993 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.070343 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.069998 6971 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.074437 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.075518 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 253. Preceding OpId from leader: term: 3 index: 254. (index mismatch)
I20260808 01:51:05.077664 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.083611 7120 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:05.083714 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.085817 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.086616 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 268: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.089272 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.098354 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 264: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.101341 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.101838 5070 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.085230 7120 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.104754 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.107785 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 1.261, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.111052 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.122223 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.122561 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 2.256, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.123970 7117 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.126622 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.127868 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 2.252, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.128193 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.132556 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.134135 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.137173 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.143596 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 252. Preceding OpId from leader: term: 2 index: 253. (index mismatch)
I20260808 01:51:05.145465 7120 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20260808 01:51:05.147603 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.152586 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.153745 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.155582 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.161485 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20260808 01:51:05.162802 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 261. Preceding OpId from leader: term: 1 index: 262. (index mismatch)
I20260808 01:51:05.158068 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20260808 01:51:05.165645 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:05.158767 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.167284 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.168190 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:05.174897 7120 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:05.175307 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.178393 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.180716 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.253: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.189879 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.190630 7117 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.193300 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.202541 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 1 index: 266. (index mismatch)
I20260808 01:51:05.204453 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:05.208765 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.211712 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.218421 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.221029 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.221349 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.207486 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.226948 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 269: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.233779 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.233667 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.235173 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.257977 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 265: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.268316 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 2.253, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.275058 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20260808 01:51:05.277534 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 3.254, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.278492 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.280683 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.283769 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.284227 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 254. Preceding OpId from leader: term: 3 index: 255. (index mismatch)
I20260808 01:51:05.285729 7117 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:05.281168 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:05.285179 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.287792 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 252 to 253, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.289077 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.290162 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.292905 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20260808 01:51:05.295373 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.296052 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.298463 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:05.301827 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.301023 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.314071 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.322760 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.323199 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.327943 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 2.253, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.334514 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 1.262, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.338207 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.338656 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.339306 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:05.339452 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
W20260808 01:51:05.339679 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.340859 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:05.350076 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.351709 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.361387 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.362663 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 1 index: 256. (index mismatch)
I20260808 01:51:05.363276 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 2.270, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.369323 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 270. Preceding OpId from leader: term: 2 index: 271. (index mismatch)
I20260808 01:51:05.370820 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:05.371595 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.371937 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.374174 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 253. Preceding OpId from leader: term: 2 index: 254. (index mismatch)
I20260808 01:51:05.378780 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.379963 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.381393 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.361173 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.389640 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.390015 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.393198 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.400027 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.400552 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20260808 01:51:05.401664 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.402114 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.403419 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20260808 01:51:05.404103 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:05.404142 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.405223 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:05.405081 7120 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.406564 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:05.415917 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20260808 01:51:05.416227 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.422006 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.425148 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.427867 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.429982 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.430351 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.430285 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.435413 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.438805 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.439546 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20260808 01:51:05.438805 7008 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.441045 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20260808 01:51:05.443172 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.443325 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.451717 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:05.452030 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.453766 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:05.456717 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.456418 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.458487 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.254: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.462124 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 253: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.464102 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.476027 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.475838 7080 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.484984 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.485680 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 270: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.486991 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.489001 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.491012 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.492189 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 253 to 254, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.494580 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.496084 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.498085 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.498818 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 266: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.498966 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.502269 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.515401 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 2.271, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.519731 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 271. Preceding OpId from leader: term: 2 index: 272. (index mismatch)
I20260808 01:51:05.523052 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.001s
I20260808 01:51:05.529021 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.531148 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.531585 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.532477 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.536043 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.536834 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 3.255, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.538625 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.539346 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 2.254, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.544838 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.547338 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 271: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.552076 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20260808 01:51:05.552903 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 255. Preceding OpId from leader: term: 3 index: 256. (index mismatch)
I20260808 01:51:05.553530 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 1.264, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.554132 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:05.556298 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.556675 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.557300 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.557655 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.556972 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.001s
I20260808 01:51:05.558813 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20260808 01:51:05.560204 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20260808 01:51:05.561668 6971 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:05.563911 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 2.255, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.569039 6971 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.569190 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.570791 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.571167 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.578056 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 254. Preceding OpId from leader: term: 2 index: 255. (index mismatch)
I20260808 01:51:05.578385 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.580904 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 255. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20260808 01:51:05.582290 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:05.582942 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I20260808 01:51:05.582885 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.583992 7117 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.584160 7008 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.592042 7080 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.001s
I20260808 01:51:05.593040 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.597158 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.603430 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.603847 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.606191 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.606618 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.615163 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20260808 01:51:05.616216 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.618212 6971 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:05.622226 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20260808 01:51:05.623765 6971 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:05.624092 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.627738 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.630378 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20260808 01:51:05.630029 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.635537 7059 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.637701 7080 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:05.639778 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.640233 7117 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.641652 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.255: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.641880 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.650691 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.651647 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.651670 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.658363 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.658708 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.661914 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.663133 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 1 index: 266. (index mismatch)
I20260808 01:51:05.662643 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.662765 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.667006 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 3.256, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.667905 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.674014 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.675257 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:05.678879 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.686039 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.688081 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.689965 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.691826 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 272: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.693186 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.699148 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.699708 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.715744 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 267: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.719921 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.722450 7117 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:05.723273 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 44936.8us
I20260808 01:51:05.726130 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20260808 01:51:05.731343 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.732861 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 256. Preceding OpId from leader: term: 3 index: 257. (index mismatch)
I20260808 01:51:05.738174 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.738961 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.740329 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:05.740079 7008 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.002s
I20260808 01:51:05.745754 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.747443 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.748909 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.750113 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 254 to 255, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.751197 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.755705 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.762315 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.764374 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.768313 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 2.256, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.771972 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.774443 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.774959 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 2.255, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.778688 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20260808 01:51:05.780551 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20260808 01:51:05.781155 7120 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:05.782058 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:05.755002 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.788336 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.788689 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.789376 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.792379 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 255. Preceding OpId from leader: term: 2 index: 256. (index mismatch)
I20260808 01:51:05.785347 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.793743 7120 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20260808 01:51:05.794041 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.795552 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.800194 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.802102 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.802627 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.803624 7008 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.806703 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.256: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.804175 7117 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.813398 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.832883 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.833261 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.834826 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 255: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.838904 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.847239 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.847569 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 3.257, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.853521 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.853962 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.859817 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 257. Preceding OpId from leader: term: 3 index: 258. (index mismatch)
I20260808 01:51:05.860723 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.861856 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.862191 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.862243 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20260808 01:51:05.863341 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20260808 01:51:05.863972 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.864281 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.865069 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 255 to 256, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 256 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.866637 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:05.867970 7120 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:05.870090 7120 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:05.877197 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20260808 01:51:05.883873 7008 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:05.886824 7120 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.900013 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.900749 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:05.906097 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.906504 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.907738 7080 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.913851 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.914311 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 2.256, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.920542 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.923074 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:05.925050 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:05.925487 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.931358 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20260808 01:51:05.933272 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20260808 01:51:05.933440 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:05.927629 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
W20260808 01:51:05.933820 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:05.934314 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20260808 01:51:05.936879 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:05.938267 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:05.939010 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.939792 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:05.947543 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.952431 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.953338 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 256. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20260808 01:51:05.955060 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.962139 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:05.964425 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.969471 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:05.976269 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.978288 7008 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20260808 01:51:05.981297 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 3.258, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:05.983428 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.989017 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.990135 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:05.994992 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:05.998932 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.002504 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.008486 7120 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.015754 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.017176 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.019217 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 258. Preceding OpId from leader: term: 3 index: 259. (index mismatch)
I20260808 01:51:06.024457 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.024883 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.027758 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 273: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.028594 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 268: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.028957 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20260808 01:51:06.033424 7080 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:06.034719 7080 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:06.035910 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.037176 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.039608 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.041285 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.045753 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20260808 01:51:06.056746 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.057454 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.062906 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20260808 01:51:06.064569 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20260808 01:51:06.074048 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.257: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.080199 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:06.082962 7043 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.083178 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.086844 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 256: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.092576 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.093818 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.094512 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20260808 01:51:06.096568 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:06.117236 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:06.123416 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.136121 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.145147 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.151003 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.152446 7008 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.152590 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.158604 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.164288 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.166209 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.166703 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.166551 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.168618 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.169768 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 256 to 257, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.180306 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:06.182993 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.196903 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.202644 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.204090 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.206576 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 269: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.207013 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 1.259, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.207729 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 3.259, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.212970 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.213338 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.214668 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.215015 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.217489 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:06.220211 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20260808 01:51:06.221876 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.001s
I20260808 01:51:06.233450 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.239617 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 2.257, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.240796 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.246280 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.247931 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 259. Preceding OpId from leader: term: 1 index: 260. (index mismatch)
I20260808 01:51:06.249960 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.253609 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 274: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.250288 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:06.255501 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 259. Preceding OpId from leader: term: 3 index: 260. (index mismatch)
I20260808 01:51:06.256860 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20260808 01:51:06.265008 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:06.265937 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.267148 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.268049 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20260808 01:51:06.268561 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.289649 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.291685 7120 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.292809 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.293565 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 257. Preceding OpId from leader: term: 2 index: 258. (index mismatch)
I20260808 01:51:06.304587 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:06.304850 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.306448 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:06.309090 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.309535 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20260808 01:51:06.309918 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.311419 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.313946 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.318418 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.323549 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.318450 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.323798 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.258: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.323573 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 2.262, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.325950 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.332876 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 257: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.335649 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:06.337713 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20260808 01:51:06.338996 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 262. Preceding OpId from leader: term: 2 index: 263. (index mismatch)
I20260808 01:51:06.340468 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:06.340754 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:06.348791 7120 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.350075 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.353132 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.353430 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.359077 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.361935 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.367615 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.361191 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.370996 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 257 to 258, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.372524 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.394217 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.404300 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.409439 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:06.409495 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
W20260808 01:51:06.410391 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.411070 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:06.418903 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.421490 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 1.260, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.417798 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20260808 01:51:06.426731 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 260. Preceding OpId from leader: term: 1 index: 261. (index mismatch)
I20260808 01:51:06.427786 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 2.258, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.427522 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.428174 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:06.428805 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:06.431851 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.434984 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.435360 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.435773 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 258. Preceding OpId from leader: term: 2 index: 259. (index mismatch)
I20260808 01:51:06.436652 7120 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.438195 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.445154 7080 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.001s
I20260808 01:51:06.445847 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.443394 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.453321 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.454857 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.259: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.459200 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.461274 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20260808 01:51:06.463979 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.465080 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:06.468731 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.469094 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.471289 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20260808 01:51:06.472702 7120 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:06.474861 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.479209 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.488646 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.490298 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.497994 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.498518 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.504408 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 270: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.507555 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.508701 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 275: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.479651 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.511965 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.518831 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.531972 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.534206 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.535869 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.537264 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 258 to 259, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.563367 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.568602 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.577735 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.586422 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20260808 01:51:06.588222 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:06.595036 7120 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.596200 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 1.261, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.597162 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.599844 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.602581 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20260808 01:51:06.604894 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:06.607295 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.607860 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.608229 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.612056 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.612937 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 261. Preceding OpId from leader: term: 1 index: 262. (index mismatch)
I20260808 01:51:06.618422 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20260808 01:51:06.618494 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.619793 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.626304 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.626672 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.628947 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 259, Committed index: 259, Last appended: 2.259, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.615374 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.630736 7080 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:06.623368 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:06.637665 7080 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.641109 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.643123 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.648789 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.654246 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 3.260, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.657181 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.659904 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 260. Preceding OpId from leader: term: 3 index: 261. (index mismatch)
I20260808 01:51:06.660784 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.666571 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 259. Preceding OpId from leader: term: 2 index: 260. (index mismatch)
I20260808 01:51:06.661247 7120 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:06.664651 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.671031 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.674235 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
I20260808 01:51:06.674254 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 260, Last known committed idx: 259, Time since last communication: 0.000s
I20260808 01:51:06.685206 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 271: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.687543 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.694545 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.695715 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.696106 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.698415 7120 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:06.701843 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.701510 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.699847 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20260808 01:51:06.706717 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.714265 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.716583 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.260: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.727495 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:06.731487 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 2 index: 277. (index mismatch)
I20260808 01:51:06.730260 7120 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.733286 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.733716 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.737244 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 259: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.739588 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.752823 7080 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.765048 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20260808 01:51:06.769596 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.779801 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.781469 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.798859 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 276: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.801110 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.750854 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.809520 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.811314 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.812215 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.813225 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.838579 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 1.262, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.843009 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 262. Preceding OpId from leader: term: 1 index: 263. (index mismatch)
I20260808 01:51:06.842795 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.844732 7080 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:06.848089 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:06.852998 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.856528 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20260808 01:51:06.856371 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.858105 7080 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:06.860473 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:06.860232 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:06.860918 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.861802 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:06.862172 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
W20260808 01:51:06.862298 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.863435 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:06.869841 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:06.870249 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch)
I20260808 01:51:06.871187 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:06.872418 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:06.872618 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.873592 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:06.877415 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.881902 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.884014 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.885634 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.894388 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.899204 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.910677 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.928341 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 259 to 260, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 260 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.938009 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:06.939900 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:06.946234 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.948550 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 272: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:06.941520 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:06.972095 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 2.277, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.979197 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:06.980039 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 277. Preceding OpId from leader: term: 2 index: 278. (index mismatch)
I20260808 01:51:06.981477 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:06.988148 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.991127 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20260808 01:51:06.992694 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:06.991272 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.002914 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 277: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.004604 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.004947 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:06.990091 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.019703 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 2.260, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.027160 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 2.262, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.030449 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.032032 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.033607 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.041239 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 3.261, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.049019 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 262. Preceding OpId from leader: term: 2 index: 263. (index mismatch)
I20260808 01:51:07.049360 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 261. Preceding OpId from leader: term: 3 index: 262. (index mismatch)
I20260808 01:51:07.051318 7080 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:07.052640 7120 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:07.054478 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 260. Preceding OpId from leader: term: 2 index: 261. (index mismatch)
I20260808 01:51:07.056113 7120 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20260808 01:51:07.058405 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.063434 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.072149 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.073501 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.075284 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.077575 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.273, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.077414 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.081651 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 274. (index mismatch)
I20260808 01:51:07.081787 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.082645 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.083123 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:07.087697 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch)
I20260808 01:51:07.090641 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:07.096319 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.101397 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.107749 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.116098 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 2.262, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.112462 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.120040 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 2.265, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.120558 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 1.263, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.122762 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.123627 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.122103 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.130244 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.136351 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:07.139211 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 2.278, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.142055 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 273: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.141417 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:07.143532 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.148104 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.150430 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.150943 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.156868 7080 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.153214 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 3.262, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.144649 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.146867 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 265. Preceding OpId from leader: term: 2 index: 266. (index mismatch)
I20260808 01:51:07.148279 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 262. Preceding OpId from leader: term: 2 index: 263. (index mismatch)
I20260808 01:51:07.150674 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 263. Preceding OpId from leader: term: 1 index: 264. (index mismatch)
I20260808 01:51:07.168217 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 278. Preceding OpId from leader: term: 2 index: 279. (index mismatch)
I20260808 01:51:07.169574 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:07.170701 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.171149 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.172691 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.173473 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.174993 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.178272 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.178738 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.184125 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.185786 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.187302 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:07.188620 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:07.193459 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 262. Preceding OpId from leader: term: 3 index: 263. (index mismatch)
I20260808 01:51:07.197374 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 278: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.198731 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.261: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.202683 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.198753 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.204159 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.273, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.207479 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 260: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.209232 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.209666 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.213284 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:07.213382 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
W20260808 01:51:07.213729 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
I20260808 01:51:07.219409 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
Flags of checked categories for Master:
I20260808 01:51:07.219188 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 260 to 261, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
I20260808 01:51:07.222316 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
I20260808 01:51:07.229755 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 274. (index mismatch)
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 258 | Yes
B | A B* | 2 | 257 | Yes
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
I20260808 01:51:07.236991 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:07.239257 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.241721 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.243180 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
I20260808 01:51:07.247303 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
I20260808 01:51:07.252235 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.252667 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
I20260808 01:51:07.260871 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 3 | 260 | Yes
B | A B* | 3 | 259 | Yes
Tablet 24ad2932d9a1489b96a63bc762f6a23f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:07.263206 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:07.263552 7080 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
I20260808 01:51:07.265563 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.266923 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.269712 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.271471 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
I20260808 01:51:07.276574 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 263 | Yes
B | A B* C~ | 2 | 262 | Yes
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:07.287685 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.288766 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 275 | Yes
B | A* B | 2 | 274 | Yes
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 1 | 260 | Yes
B | A B* | 1 | 259 | Yes
C | [config not available] | | |
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* | 2 | 259 | Yes
B | A B* C~ | 2 | 258 | Yes
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* | 2 | 261 | Yes
B | A B* C~ | 2 | 260 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 3 | 0 | 0 | 7
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 4
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 24
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:07.297644 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.299077 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.298974 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.302254 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20260808 01:51:07.306324 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 274: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.306497 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.307109 7120 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20260808 01:51:07.308063 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.315276 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.301323 7080 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.319588 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.326840 7120 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.335723 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.346375 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.346829 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.348968 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.350077 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.350198 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.353416 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 261: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.354244 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
I20260808 01:51:07.348233 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.362165 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:07.362800 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.366686 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 2.266, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.369678 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.378899 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.380831 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 1.275, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.380870 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.383222 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 2.279, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.382519 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 261 to 262, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.384428 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.388305 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 266. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20260808 01:51:07.389168 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
I20260808 01:51:07.390789 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:07.392997 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 279. Preceding OpId from leader: term: 2 index: 280. (index mismatch)
I20260808 01:51:07.398172 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:07.398799 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.403115 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:07.404886 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.405215 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.407182 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.412617 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:07.403120 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 275. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20260808 01:51:07.416572 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 279: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.417582 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:07.419479 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.420682 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.422729 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.424296 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.424851 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.432533 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.439042 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.440647 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 1.264, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.444020 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:07.445876 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 274 but the committed config has opid_index of 275
I20260808 01:51:07.448014 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.453084 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20260808 01:51:07.455305 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.457674 7156 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:07.458762 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 3.263, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.458866 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 2.262, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.464936 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.465309 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.466388 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 262. Preceding OpId from leader: term: 2 index: 263. (index mismatch)
I20260808 01:51:07.466606 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.468122 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 263. Preceding OpId from leader: term: 3 index: 264. (index mismatch)
I20260808 01:51:07.468587 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.469969 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I20260808 01:51:07.470067 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.471175 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:07.471242 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.478788 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.478372 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.479523 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 275: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.490036 7156 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.491366 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.491612 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.263: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.494520 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.496261 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.500278 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 1.275, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.499616 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 262: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.497730 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.504693 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.505949 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 2.280, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.506032 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.512574 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.512984 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.514573 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 275. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20260808 01:51:07.517007 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:07.517719 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 280. Preceding OpId from leader: term: 2 index: 281. (index mismatch)
I20260808 01:51:07.518664 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.519078 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.519579 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:07.521768 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 262 to 263, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 263 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.523167 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.525425 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.526578 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.527769 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.524755 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.535924 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20260808 01:51:07.536448 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.538179 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:07.538537 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20260808 01:51:07.539682 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20260808 01:51:07.541920 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 1.276, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.542302 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.543124 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:07.544750 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.546044 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.546406 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.549185 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.551021 7120 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:07.551791 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.552178 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.553498 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.556361 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.559332 7120 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.560292 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 280: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.562644 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.563089 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.565448 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 276. Preceding OpId from leader: term: 1 index: 277. (index mismatch)
I20260808 01:51:07.567489 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.001s
I20260808 01:51:07.573288 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 3.264, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.572788 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.576452 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.576707 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.578835 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 263, Committed index: 263, Last appended: 2.263, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.584343 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 264. Preceding OpId from leader: term: 3 index: 265. (index mismatch)
I20260808 01:51:07.589030 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.589046 7158 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.001s
I20260808 01:51:07.590085 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.590499 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.593070 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.593427 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.594439 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.599047 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.600930 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 2 index: 264. (index mismatch)
I20260808 01:51:07.600772 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.600824 7158 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.602169 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 264, Last known committed idx: 263, Time since last communication: 0.000s
I20260808 01:51:07.604377 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 1 index: 266. (index mismatch)
I20260808 01:51:07.606195 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.602691 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 276: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.607280 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.607944 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.607807 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.609400 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.612646 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.614490 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.624203 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.624198 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.625759 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.625882 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.629019 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.630585 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 2.281, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.633544 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.264: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.636636 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 263: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.636873 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20260808 01:51:07.636530 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 1.276, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.638628 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:07.640027 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 2.268, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.641659 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 276. Preceding OpId from leader: term: 1 index: 277. (index mismatch)
I20260808 01:51:07.644006 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20260808 01:51:07.648142 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.650339 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20260808 01:51:07.650115 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 2.265, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.652251 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.656394 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.656783 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.658016 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.658504 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.001s
I20260808 01:51:07.657348 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.659174 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.660492 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.666503 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 265. Preceding OpId from leader: term: 2 index: 266. (index mismatch)
I20260808 01:51:07.668851 7120 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.670783 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.672657 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 281: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.673647 7120 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.674916 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.682065 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.686532 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.693766 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.277, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.694926 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.695822 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.697943 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.696870 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 263 to 264, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 264 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.700420 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.702000 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 278. (index mismatch)
I20260808 01:51:07.704443 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:07.710084 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.710722 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.711740 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.712261 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.719951 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 277: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.723147 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.726315 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 2.264, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.729662 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20260808 01:51:07.730227 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 264. Preceding OpId from leader: term: 2 index: 265. (index mismatch)
I20260808 01:51:07.731583 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 3.265, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.733211 7120 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:07.733388 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20260808 01:51:07.739892 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.740581 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.741271 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.742470 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.746330 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.746721 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.745568 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20260808 01:51:07.748543 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.750066 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.749193 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.751423 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 265. Preceding OpId from leader: term: 3 index: 266. (index mismatch)
I20260808 01:51:07.735440 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 2.265, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.752269 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.757730 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.758088 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.761596 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.761556 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:07.763166 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.763702 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.265: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.766429 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 265. Preceding OpId from leader: term: 2 index: 266. (index mismatch)
I20260808 01:51:07.768000 7120 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.769543 7120 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.779918 7157 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.783862 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 264: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.785154 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.786784 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.788739 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.793728 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.277, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.797704 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 1.278, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.805492 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.805895 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.805534 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.808243 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.811530 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.811869 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.813123 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.813479 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.813758 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 2.266, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.814702 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20260808 01:51:07.816831 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:07.818311 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 278. (index mismatch)
I20260808 01:51:07.819135 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20260808 01:51:07.819533 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 266. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20260808 01:51:07.819587 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:07.823009 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20260808 01:51:07.823817 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:07.823433 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.825387 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 282: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.829986 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.833072 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.839860 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.839967 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.839751 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.842288 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.843566 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.843640 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.848022 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.846719 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.847729 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.854249 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.860682 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.865897 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.868352 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.873435 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 278: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.873935 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 264 to 265, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:07.875270 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.851477 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.884415 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.912144 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.916319 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 2.265, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:07.916486 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 3.266, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.923928 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 266. Preceding OpId from leader: term: 3 index: 267. (index mismatch)
I20260808 01:51:07.924448 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20260808 01:51:07.926365 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:07.928650 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 265. Preceding OpId from leader: term: 2 index: 266. (index mismatch)
I20260808 01:51:07.927467 7159 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
W20260808 01:51:07.929754 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.930692 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:07.931075 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:07.907884 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.912914 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 2.266, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.931630 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20260808 01:51:07.932082 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:07.940243 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.942842 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.943419 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.946622 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.266: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.910211 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 2.270, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:07.957401 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 270. Preceding OpId from leader: term: 2 index: 271. (index mismatch)
I20260808 01:51:07.958871 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:07.965898 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.971599 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:07.977516 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.979324 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:07.984620 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:07.990612 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 266. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20260808 01:51:07.997141 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:07.998333 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 265: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:07.997753 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.000905 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.001516 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.002765 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.001s
I20260808 01:51:08.007052 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.010521 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.016484 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20260808 01:51:08.019066 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.019454 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.022598 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 1.278, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.024286 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.026139 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 265 to 266, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 266 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.027307 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
I20260808 01:51:08.027305 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.029047 7160 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.001s
I20260808 01:51:08.029886 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.033895 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.039050 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.039875 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20260808 01:51:08.040406 7157 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.001s
I20260808 01:51:08.044896 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20260808 01:51:08.046445 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.047016 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:08.047689 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.054105 7160 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.001s
I20260808 01:51:08.059024 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.060107 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.064550 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.067452 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.073179 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.080510 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 283: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.084009 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 2.266, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.084722 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.084792 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 3.267, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.087829 7160 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.088871 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.093477 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 267. Preceding OpId from leader: term: 3 index: 268. (index mismatch)
I20260808 01:51:08.095347 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.095939 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.096884 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 2.271, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.098266 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:08.100768 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 266. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20260808 01:51:08.101580 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.101963 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.097209 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.104348 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.108209 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20260808 01:51:08.095347 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.112186 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 279: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.118654 7120 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20260808 01:51:08.119589 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.118067 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.005s
I20260808 01:51:08.126163 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 271. Preceding OpId from leader: term: 2 index: 272. (index mismatch)
I20260808 01:51:08.126348 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.127882 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.130194 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.130611 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.133687 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:08.135345 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.137804 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.139858 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.267: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.143792 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.144280 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.146997 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 2.268, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.147166 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.150236 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.158455 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20260808 01:51:08.158455 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20260808 01:51:08.159301 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.159689 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.151013 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.160342 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:08.160853 7157 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.001s
I20260808 01:51:08.161384 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.161275 7158 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.164964 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 266: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.166887 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.167340 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.172200 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20260808 01:51:08.171304 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.175158 7160 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.176396 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.176824 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.174866 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.012s
I20260808 01:51:08.188665 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.189566 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.193324 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.195520 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.195873 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.199229 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.199708 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.200490 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20260808 01:51:08.203379 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.206089 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.001s
I20260808 01:51:08.213570 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.217535 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.218613 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.220747 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.220496 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.227069 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.235003 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.241405 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 3.268, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.244547 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 284: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.244805 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.245234 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.254664 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20260808 01:51:08.256639 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.001s
I20260808 01:51:08.257822 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.259609 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 266 to 267, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.260854 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.262399 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.263963 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.266913 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.273121 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 268. Preceding OpId from leader: term: 3 index: 269. (index mismatch)
I20260808 01:51:08.275245 7158 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.001s
I20260808 01:51:08.282828 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.284696 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.287048 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.289691 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.295028 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20260808 01:51:08.294945 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.296478 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20260808 01:51:08.296844 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:08.297958 7160 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.001s
I20260808 01:51:08.298619 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.306968 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.298000 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.314530 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.314920 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.316550 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 280: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.321700 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 2.268, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.329238 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20260808 01:51:08.331070 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.001s
I20260808 01:51:08.333150 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.339644 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.340029 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.342314 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20260808 01:51:08.348163 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.351527 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20260808 01:51:08.354130 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.357969 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.350476 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.363080 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20260808 01:51:08.365083 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.366315 7158 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.374713 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:08.381551 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 285: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.384188 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.385829 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.387494 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.393544 7002 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.405735 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.409284 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.414366 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.415409 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20260808 01:51:08.417104 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.001s
I20260808 01:51:08.417099 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.418401 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.420655 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.421011 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.423487 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.429287 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.434484 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.434949 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.435442 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 267: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.437465 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20260808 01:51:08.436745 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.439041 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20260808 01:51:08.441239 7160 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.001s
I20260808 01:51:08.443390 7160 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:08.447002 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.452687 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 267 to 268, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.454006 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.455329 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.460009 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.461923 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.474433 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.474874 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.477162 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.479918 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 3.269, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.480643 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 2.268, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.482829 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.476104 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.489329 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:08.489421 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 269. Preceding OpId from leader: term: 3 index: 270. (index mismatch)
W20260808 01:51:08.489717 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.489984 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.491228 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:08.497424 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:08.497521 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
W20260808 01:51:08.497823 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.496284 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.494812 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.499050 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:08.499946 7158 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.503108 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.509560 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.509490 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 268. Preceding OpId from leader: term: 2 index: 269. (index mismatch)
I20260808 01:51:08.507800 7160 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.518788 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20260808 01:51:08.523250 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.527427 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.001s
I20260808 01:51:08.528142 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.528581 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:08.534317 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.534695 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.536453 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20260808 01:51:08.538558 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 281: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.539353 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.543712 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.547703 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 286: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.550434 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:08.551057 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.552524 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.552860 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.555972 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.269: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.564240 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.556026 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.568877 7120 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.578785 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.579250 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.579358 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20260808 01:51:08.580922 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:08.590869 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.593353 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.579252 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.598305 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20260808 01:51:08.597798 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.599789 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:08.608773 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.613399 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.636014 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.637389 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.646634 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.648418 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.649663 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.651412 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.652841 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.654301 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.655395 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:08.667448 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.672516 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.687222 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 268 to 269, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.689808 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.691345 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 3.270, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.695065 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 2.270, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.695371 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.702488 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 2.270, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.711320 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.709003 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 2.269, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.733901 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.734314 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.828941 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.830289 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20260808 01:51:08.831864 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 270. Preceding OpId from leader: term: 2 index: 271. (index mismatch)
I20260808 01:51:08.836108 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:08.837208 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 270. Preceding OpId from leader: term: 3 index: 271. (index mismatch)
I20260808 01:51:08.838212 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20260808 01:51:08.839649 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 269. Preceding OpId from leader: term: 2 index: 270. (index mismatch)
I20260808 01:51:08.844576 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:08.846213 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 270. Preceding OpId from leader: term: 2 index: 271. (index mismatch)
I20260808 01:51:08.846896 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:08.847433 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20260808 01:51:08.847764 7120 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:08.849355 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.854398 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:08.859866 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 2.287, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:08.860919 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.864344 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.864789 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.871795 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20260808 01:51:08.872591 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:08.871819 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 287. Preceding OpId from leader: term: 2 index: 288. (index mismatch)
W20260808 01:51:08.873399 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.874096 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:08.873871 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.877821 7120 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.880445 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20260808 01:51:08.885603 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.886070 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.889942 7158 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.892248 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.892422 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.898157 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.270: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.899560 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:08.900866 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:08.901700 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.901958 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.909766 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.911900 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.913722 7160 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.917227 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.918051 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.919993 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.923045 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.929298 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.929657 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.938547 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 282: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.941094 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 287: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.948765 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:08.948503 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.956509 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:08.956884 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:08.961359 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 269: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.962978 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.963829 7158 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.965224 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.966610 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.967720 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.968897 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:08.970534 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:08.975613 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch)
I20260808 01:51:08.977366 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:08.983474 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:08.985563 7156 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:08.993467 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:08.999181 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.002348 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.002532 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.007203 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.003820 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.008139 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.011564 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20260808 01:51:09.013576 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:09.018739 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.021651 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.027485 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.030896 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 3.271, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.032454 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.039433 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.042178 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 2.271, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.042647 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.043691 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.044086 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.044122 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 269 to 270, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.047360 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20260808 01:51:09.049582 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 271. Preceding OpId from leader: term: 3 index: 272. (index mismatch)
I20260808 01:51:09.054903 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:09.054715 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
W20260808 01:51:09.055238 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.056751 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:09.058321 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.060997 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 271. Preceding OpId from leader: term: 2 index: 272. (index mismatch)
I20260808 01:51:09.066551 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:09.069204 7158 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.073062 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 2.271, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.079389 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.081449 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.084201 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 2.288, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.086437 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 2.270, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.088946 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20260808 01:51:09.090521 7059 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:09.094575 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.094949 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.098762 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.099144 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.100226 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20260808 01:51:09.099375 7156 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.102361 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:09.102952 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 271. Preceding OpId from leader: term: 2 index: 272. (index mismatch)
I20260808 01:51:09.103988 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch)
I20260808 01:51:09.106523 7157 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.001s
I20260808 01:51:09.106289 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.106986 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.110581 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.110663 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 270. Preceding OpId from leader: term: 2 index: 271. (index mismatch)
I20260808 01:51:09.113113 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.113315 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.117552 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.118846 7158 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:09.119870 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20260808 01:51:09.116679 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.124830 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.126874 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.130553 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.131381 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.134680 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.134343 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.138257 7157 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.144526 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.151733 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.151837 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.153920 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.154623 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.155256 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 288: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.155984 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 283: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.156278 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.158259 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.271: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.174923 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.180724 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.181119 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.182251 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20260808 01:51:09.183566 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:09.190665 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.189720 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.192162 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.192687 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.193749 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.194854 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.196245 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.197422 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.198734 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 270 to 271, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.199815 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.202032 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.219022 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.224035 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
I20260808 01:51:09.224236 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.225515 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:09.231081 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.235798 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 3.272, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.235813 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.236801 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.239038 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.273, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.249208 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20260808 01:51:09.249341 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 272. Preceding OpId from leader: term: 3 index: 273. (index mismatch)
I20260808 01:51:09.250270 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20260808 01:51:09.250551 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 274. (index mismatch)
I20260808 01:51:09.251335 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:09.251422 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.252575 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I20260808 01:51:09.253540 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:09.255352 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 2.271, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.260986 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.262821 7157 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.001s
W20260808 01:51:09.265251 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.265784 7157 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:09.267009 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 2 index: 277. (index mismatch)
I20260808 01:51:09.269382 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:09.269208 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
W20260808 01:51:09.269762 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.269912 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.278331 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20260808 01:51:09.281523 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:09.279294 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.291769 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.291131 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.291957 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.300050 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.304873 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.305265 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.310720 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.318109 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 271. Preceding OpId from leader: term: 2 index: 272. (index mismatch)
I20260808 01:51:09.319005 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 3.273, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.320132 7156 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.321693 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.323707 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20260808 01:51:09.325191 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:09.321796 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.333514 7059 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.334281 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.334637 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.343240 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.336277 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.340348 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.346877 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.348176 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.351872 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 284: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.354900 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.355460 7158 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.357180 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I20260808 01:51:09.356812 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.360680 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 289: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.362620 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.382916 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 273. Preceding OpId from leader: term: 3 index: 274. (index mismatch)
I20260808 01:51:09.384896 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.396121 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:09.398346 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.399040 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.400060 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.406303 7156 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.408385 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.272: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.419411 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.420900 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.432272 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 271: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.423743 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.441360 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.441980 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.446012 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20260808 01:51:09.447456 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:09.448860 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.455516 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.461548 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 271 to 272, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.463889 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.470935 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.472518 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.474864 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.475227 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.474082 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20260808 01:51:09.477264 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:09.485674 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.487440 7059 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.489852 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.491909 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.494843 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.497989 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.516239 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 2.290, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.513309 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.520974 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 285: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.527863 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.528270 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.539655 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 2.272, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.542172 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 290. Preceding OpId from leader: term: 2 index: 291. (index mismatch)
I20260808 01:51:09.542426 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.543043 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 3.274, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.546146 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.001s
I20260808 01:51:09.548919 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.556114 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 2.277, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.561311 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 1.274, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.561196 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.562481 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.567904 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 272. Preceding OpId from leader: term: 2 index: 273. (index mismatch)
I20260808 01:51:09.568171 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 274. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
I20260808 01:51:09.569383 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 277. Preceding OpId from leader: term: 2 index: 278. (index mismatch)
I20260808 01:51:09.571455 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.574088 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:09.574461 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 274. Preceding OpId from leader: term: 3 index: 275. (index mismatch)
W20260808 01:51:09.574476 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.575055 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:09.575964 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.576349 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.580991 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:09.583653 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.001s
I20260808 01:51:09.584082 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:09.586107 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20260808 01:51:09.586864 7159 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:09.585269 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:09.588110 7059 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:09.591928 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.594452 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20260808 01:51:09.600136 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.600785 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:09.602804 7157 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.604255 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.607360 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.607494 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.621903 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.623432 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.623805 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.628377 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:09.633352 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.633589 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.273: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.636149 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 290: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.636926 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.637710 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.638916 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.643671 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.649236 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20260808 01:51:09.649999 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.650362 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.648454 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.652751 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:09.654621 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.656471 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.659919 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 1.275, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.666049 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.666471 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.666579 7158 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.671114 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20260808 01:51:09.671803 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:09.673519 5117 debug-util.cc:398] Leaking SignalData structure 0x7b0800080780 after lost signal to thread 4988
I20260808 01:51:09.675206 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.683383 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.689193 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 286: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.692601 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.696826 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:09.704322 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 275. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20260808 01:51:09.709187 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.710990 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.711614 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:09.714500 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.726418 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.727288 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.733105 7157 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.743695 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.744741 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.747490 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 272 to 273, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.752800 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.756686 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.754225 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.764046 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.768764 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.774098 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 2.291, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.778887 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.779557 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.273, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.782007 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20260808 01:51:09.785241 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.792513 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20260808 01:51:09.792608 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 291. Preceding OpId from leader: term: 2 index: 292. (index mismatch)
I20260808 01:51:09.794617 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:09.794903 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.795429 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.795807 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.788717 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.795936 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:09.796218 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:09.797370 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.798787 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20260808 01:51:09.803435 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:09.803828 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:09.805279 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.812386 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.818054 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.825145 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.826682 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 274. (index mismatch)
I20260808 01:51:09.831125 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20260808 01:51:09.836819 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.833587 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.844741 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.847079 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.851398 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 287: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.853327 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.856403 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.859550 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.864986 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.867025 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 3.275, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.868208 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 2.278, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:09.874238 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.874815 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.881472 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.881865 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.882527 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.883972 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.884367 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.885432 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 278. Preceding OpId from leader: term: 2 index: 279. (index mismatch)
I20260808 01:51:09.886981 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 275. Preceding OpId from leader: term: 3 index: 276. (index mismatch)
I20260808 01:51:09.890808 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:09.894274 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.274: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.895910 7159 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:09.896585 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
W20260808 01:51:09.902189 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.903663 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 291: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.906880 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20260808 01:51:09.908686 7158 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.001s
I20260808 01:51:09.908891 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.910226 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:09.912563 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.915970 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.923466 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.929844 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.930733 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20260808 01:51:09.931918 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.932307 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:09.933744 7059 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:09.934362 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.940719 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.947331 7157 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.952658 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.952378 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.939754 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:09.962090 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:09.963418 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.967243 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 273: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.968027 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:09.977126 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 273 to 274, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:09.978832 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:09.991190 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:09.999514 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:09.999886 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.004648 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 1.276, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.005733 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 292: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.006982 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20260808 01:51:10.014562 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20260808 01:51:10.014068 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 276. Preceding OpId from leader: term: 1 index: 277. (index mismatch)
I20260808 01:51:10.016363 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.018115 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20260808 01:51:10.018795 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.019459 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:10.024832 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:10.027683 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.029212 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.030462 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.040009 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.041608 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20260808 01:51:10.041471 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.041929 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.043036 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.043376 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:10.044178 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.049017 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.050196 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.056500 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20260808 01:51:10.056470 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.058197 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20260808 01:51:10.058702 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.063601 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 293: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.063192 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.066270 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.067484 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.072441 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 1.289, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.074832 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.076442 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 288: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.079442 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.079818 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.079931 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch)
I20260808 01:51:10.081399 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:10.096434 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 274: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.098194 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.099591 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.100773 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 274 to 275, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.101649 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.102861 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.105798 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 2.279, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.112166 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 3.276, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.112505 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.116806 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.127609 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.127969 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.136121 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.141562 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.146411 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.147694 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 279. Preceding OpId from leader: term: 2 index: 280. (index mismatch)
I20260808 01:51:10.149765 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:10.155117 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 1.289, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.152863 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.159799 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 276. Preceding OpId from leader: term: 3 index: 277. (index mismatch)
I20260808 01:51:10.160512 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.160912 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.162177 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 2 index: 277. (index mismatch)
I20260808 01:51:10.163681 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch)
I20260808 01:51:10.165179 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:10.167284 7158 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.001s
I20260808 01:51:10.169070 7158 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20260808 01:51:10.179531 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.181777 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20260808 01:51:10.187193 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.193500 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:10.196380 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 1.290, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.205904 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.206470 7158 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.208633 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.209893 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.216920 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 2 index: 277. (index mismatch)
I20260808 01:51:10.216413 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.218341 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch)
I20260808 01:51:10.219826 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:10.219300 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.223145 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20260808 01:51:10.223688 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.222028 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.226423 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.226490 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 289: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.232111 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.235826 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.238816 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.238746 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.241514 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.246433 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.246260 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.250257 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.253170 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 294: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.256986 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.277, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.263736 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.264135 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.264968 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.267871 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 278. (index mismatch)
I20260808 01:51:10.269436 7158 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:10.277328 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.279678 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.280988 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.283938 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:10.283176 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:10.284307 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.286455 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20260808 01:51:10.288139 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20260808 01:51:10.291733 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.294045 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.309034 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.315768 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 275: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.313373 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 1.290, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.324807 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.326191 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.327579 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.328733 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.331060 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.340909 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.341305 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.344214 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch)
I20260808 01:51:10.358321 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:10.361081 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.366345 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.367938 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.371752 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.380980 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 275 to 276, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.388631 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 290: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.399549 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.403244 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.404036 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 2 index: 296. (index mismatch)
I20260808 01:51:10.405580 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:10.412221 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.413594 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.425148 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.425566 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.427088 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20260808 01:51:10.428572 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:10.432516 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 295: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.434913 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.435151 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.439067 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.458052 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.475096 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 1.278, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.476315 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.477890 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 2.277, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.479301 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 2.280, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.482726 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.486591 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 3.277, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.491257 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 280. Preceding OpId from leader: term: 2 index: 281. (index mismatch)
I20260808 01:51:10.493377 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:10.495282 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:10.495338 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 278. Preceding OpId from leader: term: 1 index: 279. (index mismatch)
W20260808 01:51:10.495643 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.497025 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.497346 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.496658 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.498791 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.499265 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:10.499853 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 277. Preceding OpId from leader: term: 2 index: 278. (index mismatch)
I20260808 01:51:10.501762 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 277. Preceding OpId from leader: term: 3 index: 278. (index mismatch)
I20260808 01:51:10.514153 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:10.515777 7159 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:10.518630 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 2 index: 277. (index mismatch)
I20260808 01:51:10.519227 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20260808 01:51:10.520756 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:10.531814 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.535086 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.547513 7043 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.550757 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.557698 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.566929 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.568657 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.557067 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.013s
I20260808 01:51:10.577996 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.581480 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 291: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.582810 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 2.277, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.588586 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.592708 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.593745 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.595911 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.602316 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.612577 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.623250 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 277. Preceding OpId from leader: term: 2 index: 278. (index mismatch)
I20260808 01:51:10.613703 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.627972 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20260808 01:51:10.629671 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:10.630589 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:10.631254 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.277: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.638320 7156 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:10.640954 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20260808 01:51:10.646179 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
W20260808 01:51:10.645332 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.646466 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.648267 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.652639 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.654438 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.657204 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.661551 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.662335 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.667764 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.671180 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.679256 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.681161 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.682703 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.691912 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 296: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.692641 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.698555 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.701999 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.705159 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.712585 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.713037 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.714023 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 276 to 277, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 277 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.715687 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20260808 01:51:10.715287 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.716465 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.717224 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:10.718154 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.719209 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.720433 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.723435 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.725063 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.735239 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 292: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.752561 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:10.764961 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.770282 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20260808 01:51:10.771912 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:10.776551 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 2.278, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.779242 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 3.278, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.786252 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.788368 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 278. Preceding OpId from leader: term: 2 index: 279. (index mismatch)
I20260808 01:51:10.789825 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:10.790416 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 278. Preceding OpId from leader: term: 3 index: 279. (index mismatch)
I20260808 01:51:10.792411 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:10.794636 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.795001 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.796530 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.799964 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.803893 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.803923 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20260808 01:51:10.807861 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.809590 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.811224 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:10.816653 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.821812 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.823721 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.831394 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.832939 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.829685 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:10.849996 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 278 but the committed config has opid_index of 279
W20260808 01:51:10.853529 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 278 but the committed config has opid_index of 279
I20260808 01:51:10.857815 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 2.278, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.862566 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.868620 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20260808 01:51:10.870281 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:10.875437 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.877279 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.879191 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.890183 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 2.277, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.896713 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 278. Preceding OpId from leader: term: 2 index: 279. (index mismatch)
I20260808 01:51:10.898387 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:10.900571 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.901852 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.903879 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 277. Preceding OpId from leader: term: 2 index: 278. (index mismatch)
I20260808 01:51:10.904523 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.905180 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20260808 01:51:10.905937 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.907183 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 2.281, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.907474 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:10.908861 7043 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.912571 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:10.913954 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.920809 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.921238 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.919716 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.278: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.922566 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 297: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.930766 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20260808 01:51:10.930843 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 293: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.931782 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:10.932231 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:10.935902 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 277: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.937953 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:10.940269 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.940691 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:10.946175 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20260808 01:51:10.945685 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
I20260808 01:51:10.949728 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:51:10.954551 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
I20260808 01:51:10.957108 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B | 1 | 289 | Yes
B | A* B C~ | 1 | 288 | Yes
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 3 | 275 | Yes
B | A B* | 3 | 275 | No
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 2 | 292 | Yes
B | A* B C~ | 2 | 291 | Yes
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 1 | 276 | Yes
B | A B* C~ | 1 | 276 | No
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:10.965554 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 275 | Yes
B | A B* C~ | 2 | 274 | No
C | [config not available] | | |
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 275 | Yes
B | A B* | 2 | 275 | No
C | [config not available] | | |
I20260808 01:51:10.969266 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 4 | 0 | 0 | 6
I20260808 01:51:10.970269 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 6
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 26
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:10.980114 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:10.985575 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:10.988536 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:10.992352 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 2.279, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.992717 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 3.279, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:10.996439 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20260808 01:51:10.998821 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:10.999300 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:10.999663 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.001300 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 279. Preceding OpId from leader: term: 2 index: 280. (index mismatch)
I20260808 01:51:11.002751 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:11.013564 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.014072 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.017194 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.021435 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.028628 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.029037 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.032392 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 279. Preceding OpId from leader: term: 3 index: 280. (index mismatch)
I20260808 01:51:11.032457 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.033195 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.033885 7156 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:11.037649 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.039415 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 277 to 278, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.040776 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.041352 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.042374 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.043756 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.052477 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.056689 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.073369 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.077252 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 2.298, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.077296 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.080410 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20260808 01:51:11.081900 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:11.084765 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.085139 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.082000 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.086122 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.086346 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.090377 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20260808 01:51:11.090150 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 2.279, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.090312 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.091364 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 2.278, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.092144 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:11.097260 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 278. Preceding OpId from leader: term: 2 index: 279. (index mismatch)
I20260808 01:51:11.097445 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20260808 01:51:11.098636 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.098985 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.100499 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I20260808 01:51:11.101358 7043 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:11.109961 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 279. Preceding OpId from leader: term: 2 index: 280. (index mismatch)
I20260808 01:51:11.111235 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:11.116010 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.116415 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.118844 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.119678 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20260808 01:51:11.121227 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.127593 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.129871 7158 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.133842 7157 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.135510 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.138172 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 3.280, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.140156 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.135543 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 2.280, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.143728 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20260808 01:51:11.144595 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:11.145167 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.151538 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.155967 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.157408 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 280. Preceding OpId from leader: term: 2 index: 281. (index mismatch)
I20260808 01:51:11.158633 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.279: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.162282 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 280. Preceding OpId from leader: term: 3 index: 281. (index mismatch)
I20260808 01:51:11.168524 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.170652 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.171308 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.171695 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.176205 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:11.177461 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:11.179524 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.179893 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.180653 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:11.183063 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.190059 7157 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.199857 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.215294 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.216074 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.157565 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.224942 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.227275 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.228835 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.237145 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.242192 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 2.281, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.244664 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.251127 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20260808 01:51:11.252128 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.252554 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.252933 7043 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:11.259310 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.260131 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.263490 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.265537 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.277683 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.286687 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 294: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.287921 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 278: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.288635 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.289318 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.294878 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.297603 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20260808 01:51:11.298919 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 298: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.300943 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:11.305296 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.313090 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.314651 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.320510 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.322445 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.325405 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.331974 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.335209 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20260808 01:51:11.336776 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:11.342957 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.351956 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.362460 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.365252 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 278 to 279, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 279 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.374045 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.375192 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.376298 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.382467 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 299: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.387907 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.392074 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20260808 01:51:11.393507 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:11.394433 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.400372 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 3.281, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.404784 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20260808 01:51:11.406364 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:11.408607 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.409025 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.410552 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 281. Preceding OpId from leader: term: 3 index: 282. (index mismatch)
I20260808 01:51:11.412081 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:11.413765 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.418421 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.419862 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.422180 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 2.279, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.423200 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 2.280, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.423720 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.427448 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 280. Preceding OpId from leader: term: 2 index: 281. (index mismatch)
I20260808 01:51:11.429831 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:11.438807 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.442235 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 279. Preceding OpId from leader: term: 2 index: 280. (index mismatch)
I20260808 01:51:11.442940 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:11.446233 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20260808 01:51:11.448040 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.448688 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 295: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:11.449745 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.450379 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.450747 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.452747 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.459008 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.461292 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.463153 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.280: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.465497 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20260808 01:51:11.466893 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:11.466723 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.471514 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.472967 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.473318 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.476709 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20260808 01:51:11.477813 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch)
I20260808 01:51:11.478082 7157 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:11.479307 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.480437 7157 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:11.480432 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.480923 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.482244 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.489521 7157 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.490239 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.493805 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.500557 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.499766 7158 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.501665 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 279: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.502653 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.511569 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 2.300, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.512553 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.513085 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 279 to 280, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.518360 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 300. Preceding OpId from leader: term: 2 index: 301. (index mismatch)
I20260808 01:51:11.520407 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:11.520085 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:11.520763 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.521386 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:11.522373 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.521243 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.526335 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.530550 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.535460 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.539909 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 300: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.542762 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.544675 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.553467 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 2.280, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.554296 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 3.282, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.556159 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 2.281, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.557902 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 280. Preceding OpId from leader: term: 2 index: 281. (index mismatch)
I20260808 01:51:11.559798 7158 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20260808 01:51:11.560482 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 282. Preceding OpId from leader: term: 3 index: 283. (index mismatch)
I20260808 01:51:11.562077 7158 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:11.567418 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.572053 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.572454 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.574570 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.575222 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20260808 01:51:11.576857 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:11.577492 7157 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.578815 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.281: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.581966 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20260808 01:51:11.582042 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.582517 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.583374 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:11.585707 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.587016 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.589865 7116 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.591495 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.596402 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.596904 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.598546 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.603513 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.608183 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.611311 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.612010 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.613230 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.613600 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.614759 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.618026 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20260808 01:51:11.619699 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:11.608378 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.627978 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.629889 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.633659 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.635459 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 280 to 281, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.639271 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 296: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.645527 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.651103 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 2.301, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.651863 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.648864 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.658620 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 301. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20260808 01:51:11.658828 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 1.298, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.658792 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.660329 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:11.660096 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.662875 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 298. Preceding OpId from leader: term: 1 index: 299. (index mismatch)
I20260808 01:51:11.662474 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.664390 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:11.668494 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.668843 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.668923 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20260808 01:51:11.670832 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20260808 01:51:11.673997 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.674382 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.675815 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.677987 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.679330 7159 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:11.681182 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:11.685271 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.687748 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.690014 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.694103 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.698765 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.700269 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.689507 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.703296 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20260808 01:51:11.706667 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 301: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.703020 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.710713 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.717590 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.722463 7157 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:11.723079 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 3.283, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.726812 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20260808 01:51:11.726886 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.728503 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:11.731536 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.731900 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.735625 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 2.281, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.737885 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.739521 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 283. Preceding OpId from leader: term: 3 index: 284. (index mismatch)
I20260808 01:51:11.739856 7157 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.741165 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20260808 01:51:11.740526 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.743353 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.744537 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.744395 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.745764 7158 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:11.747201 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.756531 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.758687 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 297: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.762900 7158 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:11.766390 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.769838 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.770316 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.777706 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20260808 01:51:11.779754 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.780215 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.781970 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20260808 01:51:11.782994 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 1.299, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.786769 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20260808 01:51:11.788390 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:11.792191 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.792580 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.775753 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.793836 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.794211 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.796312 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.791821 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.798841 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.799161 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.805578 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.807121 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.815652 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 299. Preceding OpId from leader: term: 1 index: 300. (index mismatch)
I20260808 01:51:11.817819 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.825165 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 1.298, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.825763 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20260808 01:51:11.825433 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.828985 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.829404 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.827196 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.830803 7158 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:11.817304 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.282: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.841776 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 281: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.842643 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.843159 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:11.843497 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.847204 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 302: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.850342 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.852242 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.853816 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.858695 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20260808 01:51:11.860733 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:11.861871 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:11.862648 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 298. Preceding OpId from leader: term: 1 index: 299. (index mismatch)
I20260808 01:51:11.863612 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.865902 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:11.863943 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:11.877737 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.879014 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.881771 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.891477 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.893486 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.895006 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.896775 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.899421 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.898509 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.901736 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.906934 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:11.909219 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20260808 01:51:11.909947 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.910545 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:11.915750 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.917212 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.919569 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.938850 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 298: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:11.940778 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 303: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:11.950996 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 281 to 282, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.955077 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.965027 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:11.966109 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 3.284, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:11.967530 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:11.969257 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.972297 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 284. Preceding OpId from leader: term: 3 index: 285. (index mismatch)
I20260808 01:51:11.973975 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:11.981755 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.984115 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:11.990440 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20260808 01:51:11.991882 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:11.994042 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:11.995723 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 1.299, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.001735 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:12.001693 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20260808 01:51:12.003378 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:12.004818 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.004539 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.004465 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.004554 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.006166 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:12.008533 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 299. Preceding OpId from leader: term: 1 index: 300. (index mismatch)
I20260808 01:51:12.010373 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20260808 01:51:12.010596 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:12.012416 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:12.012966 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20260808 01:51:12.014128 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20260808 01:51:12.018040 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.018426 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.018525 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:12.019060 7043 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20260808 01:51:12.029826 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.031471 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.040499 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.040793 7157 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.041499 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.038743 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.048666 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.049943 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.053382 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.058323 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.066346 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.071020 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.073457 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.077482 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 299: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.082106 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 304: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.094524 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.095952 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.097011 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 282: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.088457 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.115283 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.116663 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 282 to 283, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.118307 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.139886 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.144426 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 3.285, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.149549 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 2.283, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.151146 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.154814 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.157891 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.163641 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.164021 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.164513 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 283. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20260808 01:51:12.164654 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.164937 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.164906 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.165825 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20260808 01:51:12.166606 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.167689 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20260808 01:51:12.168273 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20260808 01:51:12.169209 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.169452 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20260808 01:51:12.171609 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:12.173120 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:12.182471 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 285. Preceding OpId from leader: term: 3 index: 286. (index mismatch)
I20260808 01:51:12.186157 7157 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.187467 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:12.190151 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.189905 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.195324 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.195744 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.194885 7158 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.198709 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.199538 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.200654 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.206979 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.207365 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.210366 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.217790 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 283: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.212603 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.214509 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.222637 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20260808 01:51:12.224529 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:12.225126 7002 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.226006 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20260808 01:51:12.228050 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:12.233824 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20260808 01:51:12.234869 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.235240 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.236541 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:12.236901 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.240532 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.241544 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.244676 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.249558 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.250358 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.259155 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.262379 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.267783 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.288822 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 300: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.291455 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.291303 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.284: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.297578 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.299431 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.300887 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 283 to 284, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.302385 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.306859 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.308780 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.309725 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.311615 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.327770 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20260808 01:51:12.329365 7157 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:12.342427 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.344000 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.345093 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.352747 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.360699 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.362367 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.364717 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 305: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.366084 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20260808 01:51:12.367693 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:12.370956 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.369032 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.373500 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.376219 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20260808 01:51:12.379962 7157 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.380479 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.383186 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.383512 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.388082 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20260808 01:51:12.388703 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 3.286, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.389659 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20260808 01:51:12.395787 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20260808 01:51:12.399084 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:12.400553 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.407063 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 286. Preceding OpId from leader: term: 3 index: 287. (index mismatch)
I20260808 01:51:12.407308 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.408711 7156 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.414449 7197 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.414781 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.416463 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.423799 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.426939 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.430238 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.430907 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 2.306, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.436091 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.441071 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 2.287, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.442332 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.451622 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.460521 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.465408 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.465843 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.466945 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.467303 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.468551 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 301: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.469658 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.474593 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.476079 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.477797 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 306. Preceding OpId from leader: term: 2 index: 307. (index mismatch)
I20260808 01:51:12.479147 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.479561 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.480633 7116 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:12.482638 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20260808 01:51:12.484441 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.485908 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 287. Preceding OpId from leader: term: 2 index: 288. (index mismatch)
I20260808 01:51:12.488602 7002 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:12.488654 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:12.491776 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.495532 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.498824 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.502910 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20260808 01:51:12.500222 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 284 to 285, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.507503 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.509891 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.512080 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.513289 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.515861 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 306: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.527875 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.528579 7156 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.539779 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.542903 7157 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.543084 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.550273 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.546415 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.562155 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.570804 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 2.285, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.585343 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.585745 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.588820 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20260808 01:51:12.590230 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20260808 01:51:12.590502 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.597990 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.598342 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.598584 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.603412 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.603510 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 3.287, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.609148 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.609503 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.609581 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.617121 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:12.617477 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
W20260808 01:51:12.617508 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.619004 7197 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:12.623478 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 287. Preceding OpId from leader: term: 3 index: 288. (index mismatch)
I20260808 01:51:12.625362 7157 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:12.625741 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.627648 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.628662 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20260808 01:51:12.631187 7197 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:12.638831 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.625099 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.286: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.649362 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.660679 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 285: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.668740 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.670181 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.671224 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.672178 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.672979 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 285 to 286, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.673982 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.683661 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.688277 7156 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.704177 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 302: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.706301 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.722368 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.724220 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.727123 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.729185 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.739171 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.746445 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20260808 01:51:12.746447 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20260808 01:51:12.749150 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.749511 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.750041 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.753454 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:12.760784 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.760818 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 2.307, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.768611 7043 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.770538 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.778306 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20260808 01:51:12.780293 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.001s
I20260808 01:51:12.782317 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20260808 01:51:12.784199 7197 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.001s
I20260808 01:51:12.786545 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.792330 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.793570 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.797519 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.804729 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20260808 01:51:12.804641 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.806286 7197 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:12.806795 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.810068 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.815228 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.816864 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.834712 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.733551 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 2.288, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.844201 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.845315 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 303: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.847390 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 2.287, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.850351 7197 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.854924 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.857740 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.858136 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.860759 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.861157 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20260808 01:51:12.863003 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:12.863987 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 287. Preceding OpId from leader: term: 2 index: 288. (index mismatch)
I20260808 01:51:12.865648 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:12.870788 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 307: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.868775 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.873528 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 1.305, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.872656 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.874096 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.881414 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.882035 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.883673 7157 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.891717 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.893821 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20260808 01:51:12.893071 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.895481 7118 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:12.896148 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20260808 01:51:12.898594 7043 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20260808 01:51:12.901324 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.902428 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.903026 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.904654 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.906329 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.906531 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.908270 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.912899 7197 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.914822 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.926477 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.926823 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.932557 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.932912 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.932745 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20260808 01:51:12.933467 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.933794 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.934645 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:12.937667 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:12.940354 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20260808 01:51:12.942570 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:12.942531 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 2.287, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.945550 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:12.948604 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 3.288, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.951520 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.953970 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.955365 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.956722 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.959336 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 286 to 287, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:12.960302 7116 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.961585 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.961962 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.962064 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 287. Preceding OpId from leader: term: 2 index: 288. (index mismatch)
I20260808 01:51:12.963210 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 288. Preceding OpId from leader: term: 3 index: 289. (index mismatch)
I20260808 01:51:12.962918 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:12.963425 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:12.963375 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.964542 7043 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:12.966339 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20260808 01:51:12.969734 7043 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:12.970216 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.971740 7043 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.976025 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.978863 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.981218 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.984476 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:12.984835 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:12.985428 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:12.988435 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:12.991097 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 2.288, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:12.994367 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 2.287, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:12.994619 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.005760 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 304: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.009532 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.010382 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 308: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.011968 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.012559 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20260808 01:51:13.014776 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.015123 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.015604 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 287. Preceding OpId from leader: term: 2 index: 288. (index mismatch)
I20260808 01:51:13.012559 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
I20260808 01:51:13.016119 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:13.017542 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20260808 01:51:13.017894 7002 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:13.019253 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.022459 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.034176 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.033421 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.038497 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.040189 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.045408 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 2.288, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.050094 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.057307 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.047025 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.288: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.071375 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 1.305, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.074337 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20260808 01:51:13.075701 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20260808 01:51:13.075843 7043 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.077205 7116 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:13.080417 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.084699 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20260808 01:51:13.085117 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 2.309, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.086697 7116 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:13.087976 7116 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.088598 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.092940 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 309. Preceding OpId from leader: term: 2 index: 310. (index mismatch)
I20260808 01:51:13.094542 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:13.095309 6970 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:13.095361 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.098326 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.104313 7197 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.109117 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.111107 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.115121 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.121683 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 305: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.123310 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 309: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.123960 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 287: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.125653 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.128264 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.130044 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.134435 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.140501 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.150328 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.153935 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.155148 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.156709 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.164656 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 287 to 288, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.167639 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.186316 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.198729 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.202303 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.203796 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 1.307, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.205176 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20260808 01:51:13.207139 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.207502 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.210103 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20260808 01:51:13.211521 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:13.211709 7197 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:13.213172 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.215695 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.232086 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.239449 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 2.290, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.243911 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 290. Preceding OpId from leader: term: 2 index: 291. (index mismatch)
I20260808 01:51:13.245328 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:13.240886 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.251513 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.253058 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.253595 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.250986 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 1.289, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.258030 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.258414 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.260318 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.261507 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
I20260808 01:51:13.263209 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:13.267143 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.267485 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.269208 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch)
I20260808 01:51:13.270792 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.271139 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.271390 7156 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:13.277997 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.278069 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 1 index: 308. (index mismatch)
I20260808 01:51:13.277770 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.278677 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.280159 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:13.288247 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 2.288, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.289070 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.294864 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.295866 7156 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.300554 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.296352 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.307258 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 310: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.310681 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.276921 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.315166 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 306: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.297844 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 3.289, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.322932 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 288. Preceding OpId from leader: term: 2 index: 289. (index mismatch)
I20260808 01:51:13.324477 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20260808 01:51:13.332556 7002 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.336243 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.289: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.341089 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.343254 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.364588 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.364979 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.365772 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 289. Preceding OpId from leader: term: 3 index: 290. (index mismatch)
I20260808 01:51:13.367161 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:13.379128 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.384914 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.386016 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.387764 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.389551 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.390856 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.392953 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.393209 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.396646 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.408192 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.417878 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 2.290, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.419881 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 1.290, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.420691 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 2.291, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.420905 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:13.424573 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 288 but the committed config has opid_index of 289
I20260808 01:51:13.427062 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch)
I20260808 01:51:13.429409 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:13.434223 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:13.437505 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.438910 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
W20260808 01:51:13.439152 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.440099 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.446465 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.446872 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.447365 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 290. Preceding OpId from leader: term: 2 index: 291. (index mismatch)
I20260808 01:51:13.448990 7002 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I20260808 01:51:13.451077 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.451551 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 291. Preceding OpId from leader: term: 2 index: 292. (index mismatch)
I20260808 01:51:13.454957 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 288 to 289, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.455824 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:13.456265 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 1.307, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.457118 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:13.456746 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.460852 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20260808 01:51:13.463889 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 1.308, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.464830 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 1 index: 308. (index mismatch)
I20260808 01:51:13.464653 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:13.467901 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.468588 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 308. Preceding OpId from leader: term: 1 index: 309. (index mismatch)
I20260808 01:51:13.469525 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:13.472975 7197 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:13.478585 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 2.289, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.479493 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.487107 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 3.290, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.495628 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.496012 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.490226 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.503087 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 289. Preceding OpId from leader: term: 2 index: 290. (index mismatch)
I20260808 01:51:13.476732 7197 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.513700 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.518967 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.530052 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.522578 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.536118 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.549084 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.549901 7002 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.554998 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.556043 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 307: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.549381 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.008s
I20260808 01:51:13.559422 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:13.561220 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 308 but the committed config has opid_index of 309
I20260808 01:51:13.570360 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:13.569979 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:13.570746 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.571812 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.583377 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.585268 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.590543 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.599814 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 311: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.603154 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20260808 01:51:13.606894 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.612234 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 290. Preceding OpId from leader: term: 3 index: 291. (index mismatch)
I20260808 01:51:13.615547 7157 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.616272 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.625891 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.629817 7156 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:13.631242 7156 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:13.634940 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.637849 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.290: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.640413 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 1.308, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.650473 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 1.309, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.651768 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:13.655771 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.658900 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.659284 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.661338 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 308. Preceding OpId from leader: term: 1 index: 309. (index mismatch)
I20260808 01:51:13.663108 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:13.665048 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.665445 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.668632 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.672237 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20260808 01:51:13.674428 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:13.675403 7197 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.681565 7118 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.683670 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.686601 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.704273 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.707536 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.709937 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.711607 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.723519 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 2.291, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:13.738804 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.739185 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.739670 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 291. Preceding OpId from leader: term: 2 index: 292. (index mismatch)
I20260808 01:51:13.741079 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:13.741247 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 2.290, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.744318 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 289: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.746141 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 290. Preceding OpId from leader: term: 2 index: 291. (index mismatch)
I20260808 01:51:13.747728 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:13.748104 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:13.748044 7157 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.001s
I20260808 01:51:13.749257 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20260808 01:51:13.752029 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:13.755952 7157 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.764799 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.766345 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.780200 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.785495 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.788102 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 308: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.787223 7220 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.788843 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.797879 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.800101 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.798758 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.806622 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.803355 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 312: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.803848 7220 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.811262 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 289 to 290, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.821174 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:13.824259 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.857177 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 1.310, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.858307 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 2.290, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.863812 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.864198 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 1.309, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.871630 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:13.872671 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20260808 01:51:13.874764 7220 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:13.877972 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 310. Preceding OpId from leader: term: 1 index: 311. (index mismatch)
I20260808 01:51:13.879066 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
I20260808 01:51:13.881145 7197 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:13.882510 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:13.882678 7197 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:13.890317 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 290. Preceding OpId from leader: term: 2 index: 291. (index mismatch)
I20260808 01:51:13.890697 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.895421 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.900426 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:13.900952 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.902683 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.906119 7220 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s
I20260808 01:51:13.904999 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.915874 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.915470 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.916409 7002 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.916723 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:13.926584 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.927448 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:13.928654 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.928889 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.291: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.947048 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 309: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.947611 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.957995 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:13.956938 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.984506 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:13.989660 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20260808 01:51:13.991271 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.001s
I20260808 01:51:13.998319 7220 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:13.999719 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.000590 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 3.291, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.006232 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.006583 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.010331 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 291. Preceding OpId from leader: term: 3 index: 292. (index mismatch)
I20260808 01:51:14.011696 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:14.020566 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 2.291, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.022886 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.025995 6970 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.027815 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.034212 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:14.034497 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 309: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 309 but the committed config has opid_index of 310
I20260808 01:51:14.038371 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.042052 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.045218 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20260808 01:51:14.046789 7118 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:14.051923 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.054926 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20260808 01:51:14.056416 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 291. Preceding OpId from leader: term: 2 index: 292. (index mismatch)
I20260808 01:51:14.056620 7002 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:14.058077 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:14.066141 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.066884 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.067286 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.065084 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.065243 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.077455 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 290 to 291, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.081971 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.086022 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.092010 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.092592 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.094486 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.102905 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.104560 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.107317 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.121037 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 313: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.125137 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:14.129419 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.151083 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.152932 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.172601 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.180020 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.180444 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.181437 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 1.310, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.183113 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20260808 01:51:14.184795 7220 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:14.189311 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.189721 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.192435 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 310. Preceding OpId from leader: term: 1 index: 311. (index mismatch)
I20260808 01:51:14.192234 7220 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.194305 7118 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:14.198137 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.204356 7118 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.207124 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.209858 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.214020 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.214388 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.217203 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20260808 01:51:14.218786 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:14.226094 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.225889 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.231989 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.232353 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.234097 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
I20260808 01:51:14.235719 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:14.235550 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.248111 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.264420 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.271968 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.276150 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.281073 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.293191 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 310: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.299526 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.304402 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 3.292, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.306169 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.306556 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.308346 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20260808 01:51:14.309917 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:14.316453 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.318154 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.319799 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.321429 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.322942 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.324551 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 292. Preceding OpId from leader: term: 3 index: 293. (index mismatch)
I20260808 01:51:14.324606 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.325556 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 314: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.325748 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.326910 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:14.336719 7220 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.326360 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 2.291, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.347617 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.350800 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.356307 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20260808 01:51:14.357924 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:14.360347 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.363584 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.364231 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.357545 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.373463 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.373644 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.366904 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.377833 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.390329 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20260808 01:51:14.392814 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20260808 01:51:14.393342 7156 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20260808 01:51:14.394183 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:14.398797 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.400274 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.419220 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 3.293, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.422438 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.422554 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.428140 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.428514 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.429451 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20260808 01:51:14.430256 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.430747 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 291. Preceding OpId from leader: term: 2 index: 292. (index mismatch)
I20260808 01:51:14.436378 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.436729 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.437289 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.438115 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.438427 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.440807 7159 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:14.440762 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.442382 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20260808 01:51:14.444571 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.447904 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
I20260808 01:51:14.453017 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.454152 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.454523 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.454636 7220 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.464174 7156 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.466372 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.447026 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 1.312, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.451547 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 293. Preceding OpId from leader: term: 3 index: 294. (index mismatch)
I20260808 01:51:14.456414 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.292: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.457182 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.458452 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:14.464190 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20260808 01:51:14.479569 7002 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20260808 01:51:14.480827 7002 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:14.491187 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.495833 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.497099 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 291: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.499060 7002 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.504925 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.505827 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.501289 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 291 to 292, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.511641 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.512300 7118 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.513458 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.518694 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 312. Preceding OpId from leader: term: 1 index: 313. (index mismatch)
I20260808 01:51:14.520416 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:14.523098 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.523711 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
I20260808 01:51:14.525326 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:14.538189 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.541225 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.539036 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.545181 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.539342 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.583110 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 315: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.582309 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.632333 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.634173 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 311: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.643312 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.644693 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.649305 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.649703 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20260808 01:51:14.650142 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.650432 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 2 index: 296. (index mismatch)
I20260808 01:51:14.652210 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.653053 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.660082 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.662392 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.666337 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.669487 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.670714 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.691762 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 1.312, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.693295 7235 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.001s
I20260808 01:51:14.700701 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20260808 01:51:14.695613 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.001s
I20260808 01:51:14.702879 7220 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.001s
I20260808 01:51:14.711093 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:14.712987 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.713351 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:14.717782 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.720347 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.722013 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.723201 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 312. Preceding OpId from leader: term: 1 index: 313. (index mismatch)
I20260808 01:51:14.725286 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:14.727056 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.732545 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.734704 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20260808 01:51:14.739749 7220 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.740942 7206 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:14.743764 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.744122 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.746109 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.748588 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.751310 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.752965 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.755909 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.758390 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 312: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.762974 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.764938 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.774178 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 316: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.774943 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.780174 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 292: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.783356 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.798058 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.800051 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.801827 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.815652 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 3.294, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.818605 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.819079 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.827432 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20260808 01:51:14.827432 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 294. Preceding OpId from leader: term: 3 index: 295. (index mismatch)
I20260808 01:51:14.829566 7159 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:14.829948 7220 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:14.836098 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20260808 01:51:14.840837 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:14.842379 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.847469 7156 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.853294 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.854271 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.851487 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 292 to 293, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.850287 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.860445 7220 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.860972 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.865692 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20260808 01:51:14.865594 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.867321 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.868505 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.869168 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:14.875738 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.876319 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.877470 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.869458 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.884912 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20260808 01:51:14.886811 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.001s
I20260808 01:51:14.895870 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.900740 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.903255 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 2.293, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.906539 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.907862 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20260808 01:51:14.907974 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:14.908136 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.909432 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:14.910293 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.912850 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.919972 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.926167 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.927404 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.930050 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 313: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.933604 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 293. Preceding OpId from leader: term: 2 index: 294. (index mismatch)
I20260808 01:51:14.935189 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:14.935454 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.937718 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
W20260808 01:51:14.937731 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.941735 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20260808 01:51:14.942907 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.943245 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:14.945971 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.947499 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.294: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.952976 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.955838 7220 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.956584 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:14.961004 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:14.965011 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 293: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:14.967566 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 3.295, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:14.969753 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.973788 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.975935 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:14.976387 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:14.975776 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:14.978379 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 2 index: 296. (index mismatch)
I20260808 01:51:14.978219 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:14.984385 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:14.994055 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:14.995622 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.002024 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 295. Preceding OpId from leader: term: 3 index: 296. (index mismatch)
I20260808 01:51:15.003568 7220 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:15.004766 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.314, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.010121 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.014096 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.015866 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20260808 01:51:15.015545 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.017278 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:15.029997 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.031920 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.039016 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.044266 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.047472 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.314, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.062371 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.050274 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.065495 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.071388 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:15.071031 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:15.071744 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.072430 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 293 to 294, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.083102 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20260808 01:51:15.087894 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:15.098891 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.107218 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.092742 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20260808 01:51:15.117511 7206 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:15.120047 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.121240 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.124337 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.123175 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.127451 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20260808 01:51:15.127349 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.128809 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.131392 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.131734 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.135932 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.136677 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 2.294, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.139047 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.138931 7220 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.144243 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.145534 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.150986 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20260808 01:51:15.153395 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 2 index: 296. (index mismatch)
I20260808 01:51:15.157236 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.160843 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.161216 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.162431 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.165906 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 314: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.166972 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 294. Preceding OpId from leader: term: 2 index: 295. (index mismatch)
I20260808 01:51:15.166999 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 317: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.168637 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:15.168465 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:15.171365 4770 catalog_manager.cc:5220] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 314: ChangeConfig:ADD_PEER:NON_VOTER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 314 but the committed config has opid_index of 315
I20260808 01:51:15.170796 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.175711 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.176076 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.176626 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20260808 01:51:15.182713 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20260808 01:51:15.184456 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:15.186578 7235 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.196470 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.199131 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
I20260808 01:51:15.201954 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
I20260808 01:51:15.207211 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.212615 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.213450 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.194224 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.214730 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.215238 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 293 | Yes
B | A B* C~ | 2 | 292 | Yes
C | [config not available] | | |
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:15.231127 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.233022 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
I20260808 01:51:15.237396 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.295: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 1 | 311 | Yes
B | A* B | 1 | 311 | Yes
C | [config not available] | | |
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 3 | 292 | Yes
B | A B* | 3 | 291 | Yes
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 2 | 314 | Yes
B | A* B C~ | 2 | 313 | Yes
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* | 1 | 293 | Yes
B | A B* C~ | 1 | 292 | Yes
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:15.249908 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.252585 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 292 | Yes
B | A B* | 2 | 291 | Yes
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:15.258745 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.258744 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 1.298, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 1 | 310 | Yes
B | A* B | 1 | 310 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 3 | 0 | 0 | 7
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 6
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 26
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:15.265419 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.267831 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 298. Preceding OpId from leader: term: 1 index: 299. (index mismatch)
I20260808 01:51:15.269716 7156 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:15.272295 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20260808 01:51:15.272639 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 3.296, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.273878 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:15.279202 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.281332 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:15.284324 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 2.318, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.293318 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 296. Preceding OpId from leader: term: 3 index: 297. (index mismatch)
I20260808 01:51:15.295387 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.303926 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.305868 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.307065 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.308358 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 294 to 295, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.309501 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.309938 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.316334 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20260808 01:51:15.318245 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.319489 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 318. Preceding OpId from leader: term: 2 index: 319. (index mismatch)
I20260808 01:51:15.320601 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.320566 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:15.325107 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.317436 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:15.323786 7156 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.335210 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:15.338531 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.338960 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.340588 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.341874 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.343470 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.362666 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 318: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.366998 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.368681 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.375684 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 2.298, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.375962 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.377571 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 2.295, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.382685 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20260808 01:51:15.381553 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20260808 01:51:15.384660 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 315: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.386544 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 1.316, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.389058 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.389394 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.398463 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20260808 01:51:15.400489 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:15.404217 7157 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:15.405485 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:15.406057 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.407605 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.409524 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.414657 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.415900 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 295. Preceding OpId from leader: term: 2 index: 296. (index mismatch)
I20260808 01:51:15.417232 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.421245 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20260808 01:51:15.423350 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.423956 7156 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.425874 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.431069 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 1.299, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.435407 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.437966 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.438333 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.439894 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 299. Preceding OpId from leader: term: 1 index: 300. (index mismatch)
I20260808 01:51:15.439908 7220 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.445113 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.446247 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.452271 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 1.316, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.455446 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 2.319, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.463973 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.464623 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20260808 01:51:15.465651 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch)
I20260808 01:51:15.466101 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.466485 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.467036 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.468427 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:15.469956 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:15.476375 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 3.297, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.480180 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.482458 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.482823 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.485991 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 1.317, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.485889 7206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:15.490398 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 51905.5us
I20260808 01:51:15.491804 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.492167 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.494414 7157 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.001s
I20260808 01:51:15.496803 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.500725 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.507145 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.508806 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20260808 01:51:15.510488 7236 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:15.499138 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.512825 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 295: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.513585 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 297. Preceding OpId from leader: term: 3 index: 298. (index mismatch)
I20260808 01:51:15.435925 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.519822 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 316: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.525568 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.296: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.525808 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.525682 7236 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.535816 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.539537 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.541787 7157 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.542012 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 295 to 296, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.545150 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.546149 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 1.317, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.546671 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.547086 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.548393 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 319: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.554376 7220 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.559966 7159 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.564620 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20260808 01:51:15.565793 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.567047 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.568620 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20260808 01:51:15.570034 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:15.573875 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.578313 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.580183 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.581591 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.583868 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.585196 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.586730 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.587795 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 317: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.576704 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.592152 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.602222 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.603977 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.602849 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.604664 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.615281 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.621862 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 3.298, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.623886 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.624779 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.625440 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.629988 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.634641 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 2.296, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.635847 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.639324 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20260808 01:51:15.639734 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20260808 01:51:15.641265 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:15.643465 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20260808 01:51:15.643821 7220 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.644905 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:15.646351 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.646708 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.649578 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 296. Preceding OpId from leader: term: 2 index: 297. (index mismatch)
I20260808 01:51:15.651362 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 298. Preceding OpId from leader: term: 3 index: 299. (index mismatch)
I20260808 01:51:15.651715 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
I20260808 01:51:15.653688 7206 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:15.654610 7236 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.656010 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.657927 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.658286 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.659142 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.659507 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.645009 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.664748 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.666616 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20260808 01:51:15.667903 6970 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:15.669802 7156 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.670727 7235 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.679828 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 2.298, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.681656 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20260808 01:51:15.689708 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.689535 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.690290 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20260808 01:51:15.689363 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.698493 7159 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:15.697381 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.700670 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.702907 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.702816 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20260808 01:51:15.703054 7235 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.705408 7206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.708617 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.713920 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.714872 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.725319 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.725769 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.726732 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20260808 01:51:15.728129 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:15.729311 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.297: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.730094 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 320: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.732609 7157 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.736548 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.738978 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.743947 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.754408 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 318: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.756171 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.759963 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.760599 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.762187 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.763485 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.764540 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.773597 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:51:15.776618 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 43813.3us
I20260808 01:51:15.782030 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.784164 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.789374 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.785871 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.793781 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.794152 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.794715 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.796259 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 296 to 297, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.796887 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20260808 01:51:15.778671 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.797303 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.798367 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:15.804724 7236 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.806788 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.808161 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.810743 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20260808 01:51:15.812153 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:15.818797 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.819957 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 321: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.827033 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 2.300, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.830443 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.833664 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.834749 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.838250 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20260808 01:51:15.840116 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 300. Preceding OpId from leader: term: 2 index: 301. (index mismatch)
I20260808 01:51:15.840302 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 2.298, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.842018 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
W20260808 01:51:15.842737 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 321: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 321 but the committed config has opid_index of 322
I20260808 01:51:15.843886 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:15.846190 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20260808 01:51:15.850246 7237 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.001s
I20260808 01:51:15.854099 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.853526 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.861011 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.862814 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.866413 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.878228 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.881479 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.887400 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:15.888363 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.891343 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 319: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.897086 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 2.297, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.891088 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:15.902982 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.904623 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 3.299, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.911350 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.911757 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.913264 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:15.913750 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
W20260808 01:51:15.914201 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.915058 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.915858 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.916218 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.916419 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:15.923130 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.923581 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.926774 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20260808 01:51:15.926926 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 297. Preceding OpId from leader: term: 2 index: 298. (index mismatch)
I20260808 01:51:15.927464 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 299. Preceding OpId from leader: term: 3 index: 300. (index mismatch)
I20260808 01:51:15.930517 7237 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:15.931322 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.931701 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:15.936571 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:15.939577 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20260808 01:51:15.941574 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:15.947144 7157 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:15.954497 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:15.955503 7157 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:15.957988 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.966293 7237 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20260808 01:51:15.967705 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20260808 01:51:15.970213 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.970451 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.971786 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.983489 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:15.984856 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:15.986828 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:15.987445 7236 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.993156 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:15.994065 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:15.994498 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.000054 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.003407 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 322: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.004997 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.007956 7156 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.015213 7238 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.018787 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20260808 01:51:16.022024 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.026347 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.028352 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.029202 7236 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.001s
I20260808 01:51:16.025173 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.044476 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.050041 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.054683 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.055820 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.056658 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.059692 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.050310 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.298: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.067270 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 297: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.071867 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.078315 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.087200 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
I20260808 01:51:16.088471 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 320: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.090256 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.094391 7236 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:16.104358 7236 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.113477 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.118358 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.120172 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 297 to 298, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.121270 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.122303 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 323: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.122560 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.138615 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.140394 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.143478 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 2.301, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.145545 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20260808 01:51:16.147430 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.001s
I20260808 01:51:16.150986 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 301. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20260808 01:51:16.153002 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.153364 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.153446 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.001s
I20260808 01:51:16.159726 7236 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.163832 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.154420 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.171382 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.175974 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20260808 01:51:16.176684 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20260808 01:51:16.177718 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:16.179116 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.179538 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.162547 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.181279 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.001s
I20260808 01:51:16.193249 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.195608 7159 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.159663 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 2.300, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.206231 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.208082 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.211381 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 300. Preceding OpId from leader: term: 2 index: 301. (index mismatch)
I20260808 01:51:16.213094 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:16.214447 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.220583 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.224218 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.237999 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.240523 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.204690 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.246733 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.258177 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 321: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.264406 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 2.324, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.267437 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.271045 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.277936 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.278329 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.281697 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 324. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
I20260808 01:51:16.283490 7236 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:16.272141 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.291985 7236 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.294425 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.299933 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.330442 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.346520 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.351490 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 1 index: 323. (index mismatch)
I20260808 01:51:16.353344 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:16.355391 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.362991 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.364517 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.364941 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.366564 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.385030 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 324: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.385031 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 1 index: 323. (index mismatch)
I20260808 01:51:16.386976 7206 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:16.399832 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.405480 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 2.301, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.408509 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.417933 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.420990 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.423696 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.432250 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 2.325, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.438207 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.438652 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.437197 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20260808 01:51:16.439929 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20260808 01:51:16.441421 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.001s
I20260808 01:51:16.441802 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.001s
I20260808 01:51:16.449851 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 325. Preceding OpId from leader: term: 2 index: 326. (index mismatch)
I20260808 01:51:16.450733 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.452082 7236 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.001s
I20260808 01:51:16.459079 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.460738 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.466329 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.475219 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:16.474659 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:16.475570 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.478109 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 322: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.478062 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 2.298, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.485441 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 3.300, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.487865 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.495163 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20260808 01:51:16.498986 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.501469 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 300. Preceding OpId from leader: term: 3 index: 301. (index mismatch)
I20260808 01:51:16.503394 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.001s
I20260808 01:51:16.507016 7156 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:16.513795 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.516584 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.517091 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.528411 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.531225 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 301. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20260808 01:51:16.533004 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.536237 7237 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:16.537705 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20260808 01:51:16.539043 7156 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.539139 7237 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.542929 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:16.551888 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.299: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.551783 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.554755 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 325: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.566931 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.569209 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.580106 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.581107 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.582600 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.582893 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.580999 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.584728 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.595911 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.600310 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.605365 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.612154 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 1 index: 324. (index mismatch)
I20260808 01:51:16.614326 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.614718 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.615613 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:16.615296 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.617043 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.618569 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 298 to 299, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.624415 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.626057 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.625896 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.631871 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.636651 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 2.326, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.642690 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 326. Preceding OpId from leader: term: 2 index: 327. (index mismatch)
I20260808 01:51:16.643221 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.643582 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.644685 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:16.644348 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:16.645048 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.645675 7206 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:16.651372 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 1 index: 324. (index mismatch)
I20260808 01:51:16.652935 7206 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:16.652665 7236 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.656032 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.659647 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.661672 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20260808 01:51:16.663519 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 1 index: 305. (index mismatch)
I20260808 01:51:16.665453 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:16.666869 7238 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:16.668630 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.664080 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.677170 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.679661 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.679980 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.685043 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 326: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.695964 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.700235 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.702440 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 323: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.708630 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 3.301, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.711360 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.721508 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.709420 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.723555 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.725389 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.729001 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.729359 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.729399 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.716732 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.713529 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 2.300, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.731845 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.732180 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.738485 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.740285 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20260808 01:51:16.749105 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 301. Preceding OpId from leader: term: 3 index: 302. (index mismatch)
I20260808 01:51:16.747761 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.750867 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.751197 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 1.324, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.751631 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 2.327, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.753360 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 300. Preceding OpId from leader: term: 2 index: 301. (index mismatch)
I20260808 01:51:16.756757 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20260808 01:51:16.759200 7159 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:16.761684 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:16.763937 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.766954 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 324. Preceding OpId from leader: term: 1 index: 325. (index mismatch)
I20260808 01:51:16.768448 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:16.773159 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20260808 01:51:16.774591 7156 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.779374 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.782634 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.784231 7238 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.784862 7237 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.799912 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.802495 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 1.305, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.787137 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.790858 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.811470 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.816701 7159 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:16.817454 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.820122 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 327. Preceding OpId from leader: term: 2 index: 328. (index mismatch)
I20260808 01:51:16.821496 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.822683 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:16.826409 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.828402 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.833822 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.798378 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.842584 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.849685 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20260808 01:51:16.852662 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.853080 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.851230 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.852703 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.001s
I20260808 01:51:16.860551 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.862700 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 1.324, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.870013 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.870505 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.872834 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 2.301, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.875727 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 305. Preceding OpId from leader: term: 1 index: 306. (index mismatch)
I20260808 01:51:16.877240 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 324. Preceding OpId from leader: term: 1 index: 325. (index mismatch)
I20260808 01:51:16.882038 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.882409 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.883051 7248 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:16.880165 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.884047 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.885653 7206 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:16.901835 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.906894 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.909839 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.913398 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.915288 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 299: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.918104 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 301. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20260808 01:51:16.921873 7248 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:16.929141 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.929904 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.936652 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.940148 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.941269 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 327: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.944345 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:16.951112 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.952802 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 324: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.955547 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.955958 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 1.325, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:16.965099 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:16.966233 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 3.302, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:16.968357 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:16.968750 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:16.969838 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 325. Preceding OpId from leader: term: 1 index: 326. (index mismatch)
I20260808 01:51:16.971324 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:16.975543 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.978654 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.978727 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.980376 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.980430 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.981828 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:16.983500 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 302. Preceding OpId from leader: term: 3 index: 303. (index mismatch)
I20260808 01:51:16.983114 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:16.984247 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 299 to 300, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:16.984970 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:16.985671 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:16.996193 7156 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.001259 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.010553 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.013720 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 1.325, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.017851 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 2.328, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.018466 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 325. Preceding OpId from leader: term: 1 index: 326. (index mismatch)
I20260808 01:51:17.020393 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:17.020967 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.023808 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.024197 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.025925 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 328. Preceding OpId from leader: term: 2 index: 329. (index mismatch)
I20260808 01:51:17.027472 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:17.029137 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.028950 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.034178 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.034371 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.039561 7206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.043104 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20260808 01:51:17.043031 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.046860 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.049346 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.049834 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:17.050402 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.054088 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.054461 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.058053 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20260808 01:51:17.059576 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:17.060201 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 1.326, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.065048 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 328: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.066967 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.067333 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.070465 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.073678 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.079473 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 326. Preceding OpId from leader: term: 1 index: 327. (index mismatch)
I20260808 01:51:17.079908 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.082736 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 2.300, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.084163 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.088351 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:17.091395 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.092484 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 300. Preceding OpId from leader: term: 2 index: 301. (index mismatch)
I20260808 01:51:17.045166 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.095036 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.102422 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20260808 01:51:17.106005 7236 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.108487 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20260808 01:51:17.109479 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 2.329, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.109926 6970 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:17.110848 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20260808 01:51:17.119828 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 329. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20260808 01:51:17.119937 6970 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:17.121300 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:17.123755 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.126910 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.131259 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.136431 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.301: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.145808 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.147801 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 300: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.156101 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.160240 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.164583 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.165346 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 325: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.170778 7206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.172775 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.175009 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.174849 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.183305 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.184458 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.186143 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.187487 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.192304 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.327, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.171568 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.202435 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 329: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.208525 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.208916 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.211591 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20260808 01:51:17.213111 7236 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:17.212852 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 300 to 301, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.214766 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.221045 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.222046 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.226370 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.231420 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20260808 01:51:17.233033 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:17.229717 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.239825 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.247617 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.247995 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.248369 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20260808 01:51:17.248090 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.249861 7156 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:17.251597 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.252236 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 3.303, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.255336 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.264146 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.266778 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.267138 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.267318 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 303. Preceding OpId from leader: term: 3 index: 304. (index mismatch)
I20260808 01:51:17.274900 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.275739 7156 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:17.276168 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.278031 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.277786 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.279291 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.287164 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.291625 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.316965 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.322520 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20260808 01:51:17.323918 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.324790 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 2.330, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.325991 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:17.309286 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 1.326, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.337630 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.340750 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.341167 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.341266 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 2.306, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.343115 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.347862 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.351876 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 306. Preceding OpId from leader: term: 2 index: 307. (index mismatch)
I20260808 01:51:17.353682 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:17.355202 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.357009 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.354136 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 1.307, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.363624 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 326. Preceding OpId from leader: term: 1 index: 327. (index mismatch)
I20260808 01:51:17.366046 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.366472 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.369936 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 330. Preceding OpId from leader: term: 2 index: 331. (index mismatch)
I20260808 01:51:17.371089 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.373766 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:17.373188 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:17.381424 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.381836 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.382476 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.384927 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.386420 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 307. Preceding OpId from leader: term: 1 index: 308. (index mismatch)
I20260808 01:51:17.389420 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.389595 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.390628 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.391150 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.393788 7238 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:17.393867 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.405388 6970 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.406469 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 1.329, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.388294 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 2.301, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.412288 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20260808 01:51:17.413163 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.413552 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.414178 7236 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:17.415940 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.419685 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.422921 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.388444 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.431005 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.431396 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.437110 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.437518 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.435662 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20260808 01:51:17.438365 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 301. Preceding OpId from leader: term: 2 index: 302. (index mismatch)
I20260808 01:51:17.440539 7156 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20260808 01:51:17.445178 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 326: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.440093 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.439720 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.002s
I20260808 01:51:17.449891 7206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.458360 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.466331 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.302: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.468896 6970 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.471413 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 330: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.472124 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.474609 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.479633 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.491146 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 2.331, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.498759 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.509225 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 331. Preceding OpId from leader: term: 2 index: 332. (index mismatch)
I20260808 01:51:17.510973 7206 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:17.496773 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.512908 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 301: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.513100 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.522545 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.514559 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.524806 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.525059 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.531351 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 331: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.553732 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.327, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.555996 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 1.330, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.559460 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20260808 01:51:17.561985 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.001s
I20260808 01:51:17.570312 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.578047 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 330. Preceding OpId from leader: term: 1 index: 331. (index mismatch)
I20260808 01:51:17.579524 7206 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:17.581060 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 327: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.580960 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.573215 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.582432 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.585559 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 301 to 302, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.587181 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.594465 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 2.307, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.595278 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.597786 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 1.308, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.599303 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.607986 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20260808 01:51:17.610587 7044 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.003s
I20260808 01:51:17.621358 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 2.306, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.621830 6970 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.627169 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.627825 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.630321 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.631575 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 306. Preceding OpId from leader: term: 2 index: 307. (index mismatch)
I20260808 01:51:17.632609 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.634089 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.635008 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20260808 01:51:17.635982 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:17.638073 7237 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20260808 01:51:17.637889 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.604964 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 3.304, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.644033 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:17.644105 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
W20260808 01:51:17.644465 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.646260 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:17.646224 7206 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
W20260808 01:51:17.646639 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.657857 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20260808 01:51:17.658077 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 304. Preceding OpId from leader: term: 3 index: 305. (index mismatch)
I20260808 01:51:17.660161 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 308. Preceding OpId from leader: term: 1 index: 309. (index mismatch)
I20260808 01:51:17.660211 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.662065 7206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.676203 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.684415 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.685201 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.686354 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 1.331, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.688251 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.691784 7237 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:17.693172 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:17.693907 7238 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:17.694495 7156 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:17.695510 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.696226 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.699589 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 302 to 303, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.699474 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.703452 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.707409 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:17.711485 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.717016 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 332: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.718950 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.719341 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.724009 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20260808 01:51:17.726858 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:17.726511 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.731290 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.733533 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.735235 7257 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.737567 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.740741 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.745042 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.748242 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.748152 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.753057 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.757293 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.758047 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.758577 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 2.333, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.759599 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.764667 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.771750 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 2 index: 334. (index mismatch)
I20260808 01:51:17.772300 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.773600 7128 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:17.759755 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20260808 01:51:17.774668 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.776237 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:17.777937 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.781329 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.785184 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20260808 01:51:17.785355 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.787736 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20260808 01:51:17.790062 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.795745 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.796875 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.798615 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.800822 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.802575 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.804817 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.812258 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 333: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.813851 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 328: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.804818 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:17.805464 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20260808 01:51:17.813654 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.824036 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 2.307, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.829967 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.830338 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.837347 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20260808 01:51:17.840582 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.841472 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 3.305, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.855137 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.859854 7156 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:17.860558 7237 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.861796 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.867863 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.868319 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.871835 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.875093 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 303: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.880132 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 305. Preceding OpId from leader: term: 3 index: 306. (index mismatch)
I20260808 01:51:17.883271 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.332, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.883620 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20260808 01:51:17.887429 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20260808 01:51:17.886837 7237 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.888991 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:17.896814 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.899302 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.900462 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.897658 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.879410 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.911049 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.911914 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:17.913832 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.915117 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.921108 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 1.309, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.924957 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:17.930440 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:17.931403 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:17.934271 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 309. Preceding OpId from leader: term: 1 index: 310. (index mismatch)
W20260808 01:51:17.937129 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.939297 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:17.936602 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.941421 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.951113 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.958695 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.960870 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.962460 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.963888 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 303 to 304, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:17.965536 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:17.972239 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20260808 01:51:17.973910 7237 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.001s
I20260808 01:51:17.974640 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 2.334, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:17.980078 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 2.306, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.980885 7156 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:17.986291 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 334. Preceding OpId from leader: term: 2 index: 335. (index mismatch)
I20260808 01:51:17.987430 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:17.987831 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:17.988193 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:17.989746 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 1.329, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.995712 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:17.996114 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.004163 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20260808 01:51:18.004151 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.005715 7237 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:18.007320 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.011152 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.012236 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.014709 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20260808 01:51:18.014104 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.016201 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:18.022492 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.028009 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.028795 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.042104 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.056352 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:18.057325 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.066036 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 329: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.067554 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.067991 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.071045 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20260808 01:51:18.073099 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.001s
I20260808 01:51:18.079494 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.086767 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 3.306, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.089339 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.092424 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:18.092537 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 334: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.094331 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.096309 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.098721 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20260808 01:51:18.099239 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 306. Preceding OpId from leader: term: 2 index: 307. (index mismatch)
I20260808 01:51:18.100153 7237 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20260808 01:51:18.103946 7237 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.001s
I20260808 01:51:18.107385 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.097898 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.119024 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.121191 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.120163 7237 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.134569 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 306. Preceding OpId from leader: term: 3 index: 307. (index mismatch)
I20260808 01:51:18.137115 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.305: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.142860 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.150516 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.001s
I20260808 01:51:18.168028 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.171903 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.183701 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.188934 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.191870 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.198843 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.201344 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:18.200659 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.211735 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.246870 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.247224 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 2.309, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.253836 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:18.254175 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 309. Preceding OpId from leader: term: 2 index: 310. (index mismatch)
W20260808 01:51:18.254208 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:18.255190 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 306 but the committed config has opid_index of 307
I20260808 01:51:18.256414 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:18.257184 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 2.335, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.264801 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 1.310, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.264147 7237 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.266961 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.276201 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20260808 01:51:18.277755 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:18.280005 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 2.309, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.275521 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 310. Preceding OpId from leader: term: 1 index: 311. (index mismatch)
I20260808 01:51:18.288691 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.289233 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.291749 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:18.294406 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 309. Preceding OpId from leader: term: 2 index: 310. (index mismatch)
I20260808 01:51:18.298760 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.302814 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20260808 01:51:18.304514 7236 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:18.303772 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.311762 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.314267 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.314549 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 1.330, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.325217 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 330. Preceding OpId from leader: term: 1 index: 331. (index mismatch)
I20260808 01:51:18.326671 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.327055 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.327623 7257 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.001s
I20260808 01:51:18.338953 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.341357 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.349045 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 335: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.362583 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.381223 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 330: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.396344 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.398031 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.403052 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.411000 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:18.412446 7159 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:18.412210 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.417948 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.424023 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.425436 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.426621 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 304 to 305, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 305 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.429250 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.455346 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 1.331, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.453295 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.460529 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20260808 01:51:18.465920 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.466305 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.467064 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:18.471591 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch)
I20260808 01:51:18.473914 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:18.475111 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.479547 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.479905 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.491369 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.493232 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.501024 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.511880 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.518879 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.550375 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.553120 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.567255 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.576372 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.583992 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20260808 01:51:18.586336 7237 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.001s
I20260808 01:51:18.608284 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.611994 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 3.307, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.627918 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 331: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.629379 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.635032 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 2.307, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.636081 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.650216 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.650779 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.651641 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.652009 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.665239 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.665690 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.669951 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 307. Preceding OpId from leader: term: 3 index: 308. (index mismatch)
I20260808 01:51:18.674551 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20260808 01:51:18.682513 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.682984 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.684667 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20260808 01:51:18.686435 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:18.693681 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.695799 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.705315 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 336: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.707231 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20260808 01:51:18.708686 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:18.711947 7237 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.001s
I20260808 01:51:18.715315 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:18.715692 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.717908 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.718680 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.730973 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.732848 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.734700 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.747471 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 1.336, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.746995 7237 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.751405 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.756870 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 336. Preceding OpId from leader: term: 1 index: 337. (index mismatch)
I20260808 01:51:18.758463 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:18.765242 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.770542 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.332, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.774088 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.777846 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.778234 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.780715 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 1.311, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.782074 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.785813 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20260808 01:51:18.776643 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.789978 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.790373 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.790335 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:18.799794 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.802610 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.806615 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20260808 01:51:18.809839 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.811861 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.818143 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 305: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.819182 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.819593 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.809463 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.011s
I20260808 01:51:18.828042 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.829064 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 2.337, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.831218 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.837517 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 337. Preceding OpId from leader: term: 2 index: 338. (index mismatch)
I20260808 01:51:18.839461 7257 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:18.849237 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.849259 7257 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.859367 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.858960 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.859634 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.864903 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 332: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.871086 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20260808 01:51:18.870743 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.872965 7237 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:18.873852 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 311. Preceding OpId from leader: term: 1 index: 312. (index mismatch)
I20260808 01:51:18.876673 7237 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.879837 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.882859 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.886418 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 337: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.886690 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:18.890561 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.893857 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20260808 01:51:18.895293 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:18.901177 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.902240 7278 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.906392 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.907433 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.919155 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 333: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.922935 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.924340 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.924371 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.925858 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 305 to 306, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:18.930637 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.938509 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.938951 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.947167 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.951112 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 1.337, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.956879 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:18.957265 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:18.960098 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.963179 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 2.306, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.968617 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 337. Preceding OpId from leader: term: 1 index: 338. (index mismatch)
I20260808 01:51:18.969257 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:18.970901 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:18.972225 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:18.973915 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20260808 01:51:18.976482 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 3.308, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:18.977264 7237 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:18.977174 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.983134 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:18.987118 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:18.988896 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 306. Preceding OpId from leader: term: 2 index: 307. (index mismatch)
I20260808 01:51:18.992450 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:18.997913 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:18.999130 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.000501 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20260808 01:51:19.002043 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.002228 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.004652 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.008282 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20260808 01:51:19.008299 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.012171 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.012584 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.013432 7280 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:19.016211 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 2.338, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.015792 7159 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.023456 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 338. Preceding OpId from leader: term: 2 index: 339. (index mismatch)
I20260808 01:51:19.025696 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.026074 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.026582 7257 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:19.029446 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.032470 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 308. Preceding OpId from leader: term: 3 index: 309. (index mismatch)
I20260808 01:51:19.031919 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.043717 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:19.045040 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 334: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.045392 7257 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.050158 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.051440 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.059185 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 338: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.058606 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.065374 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.307: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.066514 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 306: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.070757 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 1.312, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.067735 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.075383 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.074218 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.078473 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.080013 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 312. Preceding OpId from leader: term: 1 index: 313. (index mismatch)
I20260808 01:51:19.083630 7237 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:19.087569 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20260808 01:51:19.087108 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.090014 7237 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.002s
I20260808 01:51:19.099835 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.100306 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.107329 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.107633 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.109058 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 306 to 307, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.115067 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.116071 7279 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.116771 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.119467 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20260808 01:51:19.121182 7237 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:19.128335 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.132332 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.137558 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.144227 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.149664 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.152577 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 2.309, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.153723 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 1.338, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.155076 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.153987 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.158300 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 2.307, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.159318 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20260808 01:51:19.160030 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch)
I20260808 01:51:19.174053 7257 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:19.174541 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.174898 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.175452 7257 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:19.175849 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 309. Preceding OpId from leader: term: 2 index: 310. (index mismatch)
I20260808 01:51:19.178015 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.178349 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.179000 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:19.179720 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.181658 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.184931 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.187676 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.195446 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 307. Preceding OpId from leader: term: 2 index: 308. (index mismatch)
I20260808 01:51:19.196555 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.197191 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.198143 7257 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.201857 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 335: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.206120 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.201689 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I20260808 01:51:19.207628 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.208789 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
Master Summary
I20260808 01:51:19.216614 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.217540 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.218755 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.308: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.204355 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
I20260808 01:51:19.226545 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 310 | Yes
B | A B* | 2 | 309 | Yes
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:19.240304 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 307: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B | 1 | 335 | Yes
B | A* B C~ | 1 | 334 | Yes
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 2 | 336 | Yes
B | A* B C~ | 2 | 335 | Yes
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 1 | 331 | Yes
B | A* B | 1 | 330 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 6 | 0 | 0 | 4
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 3
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 23
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:19.262707 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.263630 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:19.258575 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.271140 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20260808 01:51:19.271210 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.273499 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.273983 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:19.274825 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.279896 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.280265 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.281967 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.282300 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.284358 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch)
I20260808 01:51:19.285019 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20260808 01:51:19.287245 7159 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:19.288702 7159 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:19.295018 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.298420 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 3.309, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.304445 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:19.305526 7279 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.305895 7278 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:19.310505 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.314232 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.303493 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.318297 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 309. Preceding OpId from leader: term: 3 index: 310. (index mismatch)
I20260808 01:51:19.320899 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:19.321049 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.329411 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.321409 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.331835 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 307 to 308, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 308 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.339326 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.341738 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.347150 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.352559 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.354741 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.362128 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 1.336, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.365267 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 2.339, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.370357 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 339. Preceding OpId from leader: term: 2 index: 340. (index mismatch)
I20260808 01:51:19.372184 7257 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:19.376884 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.379060 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 336. Preceding OpId from leader: term: 1 index: 337. (index mismatch)
I20260808 01:51:19.380849 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.368847 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.382282 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.382014 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.383847 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.384199 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.385028 7257 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:19.388083 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20260808 01:51:19.389663 7279 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:19.394109 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch)
I20260808 01:51:19.398120 7278 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.398620 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.399290 6970 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20260808 01:51:19.395872 7281 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.397644 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:19.412878 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 339. Preceding OpId from leader: term: 1 index: 340. (index mismatch)
W20260808 01:51:19.413318 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.383304 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.415992 7044 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.417981 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.418684 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:19.420289 7128 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.425246 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.430171 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.432773 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.431758 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 308: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.440292 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.442343 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 336: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.445492 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 339: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.446362 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.451452 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.457000 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.462879 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.464888 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.467375 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 1.314, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.467555 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.468072 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.471508 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 314. Preceding OpId from leader: term: 1 index: 315. (index mismatch)
I20260808 01:51:19.471742 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 3.310, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.473004 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:19.478837 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.479261 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.480930 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20260808 01:51:19.481400 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.482170 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 310. Preceding OpId from leader: term: 3 index: 311. (index mismatch)
I20260808 01:51:19.483105 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.484391 7279 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:19.484577 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 308 to 309, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.486179 7279 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:19.488492 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20260808 01:51:19.496906 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:19.503187 7237 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.503674 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.506028 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.505896 6970 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.513788 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 2.340, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.515051 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20260808 01:51:19.514269 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 1.337, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.517585 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:19.519960 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 340. Preceding OpId from leader: term: 2 index: 341. (index mismatch)
I20260808 01:51:19.520180 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.520557 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.521675 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:19.524153 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.525336 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 337. Preceding OpId from leader: term: 1 index: 338. (index mismatch)
I20260808 01:51:19.526852 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:19.532418 7281 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.535317 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.537606 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.539414 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.542119 7128 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.542881 7159 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.548996 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.546447 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.552932 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.553593 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.555696 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.548723 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.565922 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.550180 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.560595 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 309, Committed index: 309, Last appended: 2.309, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.572346 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 337: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.573998 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.574362 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.576735 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.577098 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.577502 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.583436 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.591986 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.593142 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20260808 01:51:19.594789 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 309. Preceding OpId from leader: term: 2 index: 310. (index mismatch)
I20260808 01:51:19.596125 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 340: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.600440 7159 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.001s
I20260808 01:51:19.601999 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 310, Last known committed idx: 309, Time since last communication: 0.000s
I20260808 01:51:19.602145 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.603946 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.606428 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.611901 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.613867 6970 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.618220 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.625530 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.623714 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.310: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.626912 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.629405 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.631678 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 309: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.631862 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.637491 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.635414 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.642030 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 3.311, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.645852 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.655176 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:19.656967 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
W20260808 01:51:19.658334 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.661219 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20260808 01:51:19.663386 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 1.338, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.666198 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.666774 7159 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:19.670145 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.670573 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.667393 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:19.671166 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20260808 01:51:19.672030 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.672451 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.673511 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:19.674132 7128 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
W20260808 01:51:19.674379 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.680104 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.680517 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.681952 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch)
I20260808 01:51:19.683228 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch)
I20260808 01:51:19.683653 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 311. Preceding OpId from leader: term: 3 index: 312. (index mismatch)
I20260808 01:51:19.684950 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:19.686434 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 2.341, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.681010 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 309 to 310, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 310 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.690806 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.691000 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.694033 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.701547 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20260808 01:51:19.707401 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:19.707214 7128 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.708799 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.712343 7279 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20260808 01:51:19.713071 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.714280 7237 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:19.715402 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 2.310, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.718075 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.720964 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.721367 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.724134 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.728322 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.731110 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20260808 01:51:19.735308 6970 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.001s
I20260808 01:51:19.738078 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.742035 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.749176 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 310. Preceding OpId from leader: term: 2 index: 311. (index mismatch)
I20260808 01:51:19.753414 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.756935 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.758915 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I20260808 01:51:19.759725 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.762012 7159 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.763885 7279 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.766507 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 341: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.760425 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.778173 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.791230 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.795092 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.799109 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.799901 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 338: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.797711 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.804309 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.806005 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.801357 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.810909 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.817538 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.817919 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.820369 7288 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.821091 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.823431 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.824148 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.824566 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.823259 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20260808 01:51:19.826408 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.001s
I20260808 01:51:19.830014 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 1.342, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.835878 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch)
I20260808 01:51:19.837795 7128 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:19.842648 7281 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.844062 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.846598 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20260808 01:51:19.847282 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.847723 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.847901 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.850415 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 310: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.849710 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.851838 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.855336 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
I20260808 01:51:19.854866 7287 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.001s
I20260808 01:51:19.854399 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.857482 7278 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:19.857962 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.859839 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.311: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.863497 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.867008 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.864207 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.875204 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.875733 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 339. Preceding OpId from leader: term: 1 index: 340. (index mismatch)
I20260808 01:51:19.882179 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.888621 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:19.891395 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 342: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.891625 7279 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.897050 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 1.316, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.898742 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.899682 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.896274 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.903856 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 316. Preceding OpId from leader: term: 1 index: 317. (index mismatch)
I20260808 01:51:19.905282 7287 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.001s
I20260808 01:51:19.909488 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.916690 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20260808 01:51:19.917511 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.918015 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.918759 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:19.919255 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.913977 7287 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:19.930362 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 339: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.934832 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.935668 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.940763 7278 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.947256 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:19.948376 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.948874 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.950650 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 310 to 311, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 311 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.945528 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.958043 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:19.960530 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:19.965140 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20260808 01:51:19.966856 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:19.968658 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.969030 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.974105 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 3.312, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.976722 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:19.979163 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:19.980019 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 312. Preceding OpId from leader: term: 3 index: 313. (index mismatch)
I20260808 01:51:19.983111 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:19.983484 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:19.985437 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20260808 01:51:19.985335 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.987061 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:19.989706 7278 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.001s
I20260808 01:51:19.992033 7278 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.001s
I20260808 01:51:19.985389 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20260808 01:51:19.998826 7279 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:19.999145 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.002038 7287 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.004431 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.004587 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.008311 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.013192 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.016377 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.018229 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.020572 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.020241 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.022881 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 311: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.023591 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.023135 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.033018 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.046574 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 2.343, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.050616 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 343. Preceding OpId from leader: term: 2 index: 344. (index mismatch)
I20260808 01:51:20.051429 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.052027 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:20.056654 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 311 to 312, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.057308 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 1.317, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.058233 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.058560 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.061182 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.062744 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.062621 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.065399 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.070875 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 317. Preceding OpId from leader: term: 1 index: 318. (index mismatch)
I20260808 01:51:20.071079 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20260808 01:51:20.072256 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20260808 01:51:20.072613 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.072970 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.073588 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.074000 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.075666 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:20.075604 7278 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.001s
I20260808 01:51:20.076069 7287 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:20.079030 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.079375 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.082767 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20260808 01:51:20.082226 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.084322 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:20.085237 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.090547 7281 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.092976 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.099498 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.101141 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.100519 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.112161 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 343: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.112926 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.112669 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.116837 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 340: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.114437 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.123423 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.137698 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.141932 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.141409 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.149134 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20260808 01:51:20.151546 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 1.344, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.152959 7159 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:20.150439 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.156268 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 3.313, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.158077 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20260808 01:51:20.157812 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.157871 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.159330 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.160073 7159 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20260808 01:51:20.162297 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.166920 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20260808 01:51:20.169605 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.170068 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.169854 7279 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.170647 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:20.163982 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.173601 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.177057 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 313. Preceding OpId from leader: term: 3 index: 314. (index mismatch)
I20260808 01:51:20.177091 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.179675 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:20.177789 7159 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.187170 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.188422 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.196192 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.199901 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.202209 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.205037 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 312: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.208606 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.209481 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.213902 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20260808 01:51:20.214445 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.215641 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:20.218855 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 2.318, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.219744 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.220829 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.223598 7284 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.224732 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.226418 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
I20260808 01:51:20.227397 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.229537 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 318. Preceding OpId from leader: term: 2 index: 319. (index mismatch)
I20260808 01:51:20.230842 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:20.234125 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.234542 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.238024 7288 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.237584 7159 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.004s
I20260808 01:51:20.242084 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.247524 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.252562 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.257854 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.258981 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20260808 01:51:20.259065 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.260195 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.260949 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:20.260351 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:20.261332 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.262331 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.264771 7281 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.001s
I20260808 01:51:20.276139 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.278203 7287 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.282918 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.283867 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 312 to 313, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.291021 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.294500 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.297433 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 344: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.299160 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.273308 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch)
I20260808 01:51:20.303267 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:20.305007 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.306520 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.309033 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 3.314, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.316664 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 314. Preceding OpId from leader: term: 3 index: 315. (index mismatch)
I20260808 01:51:20.318472 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:20.318868 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.322928 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20260808 01:51:20.325856 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.001s
I20260808 01:51:20.326148 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.331521 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.316339 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 2.318, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.335114 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.335507 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.337356 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.337977 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.341003 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 318. Preceding OpId from leader: term: 2 index: 319. (index mismatch)
I20260808 01:51:20.344434 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 2.345, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.349869 7279 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:20.350489 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.350916 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.353763 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 341: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.352825 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 2 index: 314. (index mismatch)
I20260808 01:51:20.359627 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.362891 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20260808 01:51:20.364130 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.366236 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.372910 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.388455 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.389307 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.387202 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.391400 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.314: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.394289 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.399389 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.399896 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.402554 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20260808 01:51:20.403046 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 2.319, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.405200 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20260808 01:51:20.409770 7287 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:20.409497 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.002s
I20260808 01:51:20.413786 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.414247 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.422344 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 313: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.426185 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.431455 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.432129 7279 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.424520 7278 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.433322 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 313 to 314, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.438367 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.445163 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.452214 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.453989 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.455757 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20260808 01:51:20.456005 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch)
I20260808 01:51:20.461802 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:20.466579 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.468891 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.472434 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.475801 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.479475 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:20.485786 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.487330 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 2.319, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.492897 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch)
I20260808 01:51:20.493292 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.493696 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.494654 7284 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:20.497439 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 2.314, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.504729 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.512766 7284 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.515067 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.517480 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 345: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.518250 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.527325 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.529804 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 314. Preceding OpId from leader: term: 2 index: 315. (index mismatch)
I20260808 01:51:20.531140 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.531522 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20260808 01:51:20.539182 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.546289 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.315: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.526188 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.572402 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.574302 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.578071 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.583751 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.585456 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.597448 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.602430 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20260808 01:51:20.604142 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:20.611038 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.613550 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.636010 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.640749 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
I20260808 01:51:20.642283 7284 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:20.647131 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.648845 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.654033 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20260808 01:51:20.653636 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.655514 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.655938 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.657469 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.657321 7284 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:20.658650 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.665355 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 314 to 315, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.666075 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 3.315, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.669399 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.671083 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.672685 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 315. Preceding OpId from leader: term: 3 index: 316. (index mismatch)
I20260808 01:51:20.674517 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.674912 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.664634 7278 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.678514 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:20.681284 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
I20260808 01:51:20.686237 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:20.689311 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER failed with leader ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 346 but the committed config has opid_index of 347
I20260808 01:51:20.690248 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:20.695130 7287 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.703786 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 1.342, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.709022 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch)
I20260808 01:51:20.710901 7294 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:20.719111 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.719502 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.723110 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.724339 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.725942 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.732625 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 342: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.705550 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.734447 7284 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.739082 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.741122 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.743258 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.746862 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:20.746866 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
W20260808 01:51:20.747259 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.745396 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.748276 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:20.749662 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.750998 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.753505 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.754571 7294 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.756649 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.765003 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 346: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.767756 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.770067 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.770740 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.775059 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20260808 01:51:20.776489 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:20.778215 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.779690 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.784329 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.789439 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20260808 01:51:20.789851 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.790222 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.790246 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.791087 7294 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.001s
I20260808 01:51:20.791460 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.792737 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.799219 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.802517 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 343: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.805347 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.808157 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.814393 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.811201 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.813198 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 2.315, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.819391 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20260808 01:51:20.824847 7294 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:20.826121 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 315. Preceding OpId from leader: term: 2 index: 316. (index mismatch)
I20260808 01:51:20.819278 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.828011 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.828789 7284 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I20260808 01:51:20.830117 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.830444 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.830411 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 2.318, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.836825 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20260808 01:51:20.839560 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.840080 7287 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:20.841037 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.844012 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.844404 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.845701 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.847836 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20260808 01:51:20.849326 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 318. Preceding OpId from leader: term: 2 index: 319. (index mismatch)
I20260808 01:51:20.850965 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.852886 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.853260 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.854875 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20260808 01:51:20.855719 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:20.856161 7278 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:20.857329 7287 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:20.857858 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.857422 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.869784 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.870529 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 315: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.870229 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.872901 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.869549 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.316: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.877640 7285 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.882304 7288 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.884646 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.885448 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 347: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.893342 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 3.316, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.897495 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.898298 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 1.344, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.903252 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20260808 01:51:20.904345 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.904717 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.904999 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:20.908118 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20260808 01:51:20.907472 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.909653 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:20.917045 7281 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.921634 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.919514 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.923514 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 315 to 316, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.923640 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.926308 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.937531 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.938973 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:20.943881 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 344: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.944692 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 316. Preceding OpId from leader: term: 3 index: 317. (index mismatch)
I20260808 01:51:20.945019 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.947628 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:20.948865 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:20.953749 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.956125 7278 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.958173 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:20.958214 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.960011 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.968032 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.971650 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.972852 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:20.973693 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:20.979212 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:20.979611 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:20.980399 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20260808 01:51:20.983058 7281 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:20.987044 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.991314 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20260808 01:51:20.990602 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:20.991452 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.992857 7278 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:20.997702 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:20.999986 7285 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.002713 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:20.998381 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.003937 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 2.316, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.007834 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.013741 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 316. Preceding OpId from leader: term: 2 index: 317. (index mismatch)
I20260808 01:51:21.015110 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I20260808 01:51:21.018430 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20260808 01:51:21.019024 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:21.019325 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
W20260808 01:51:21.019384 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.019904 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:21.020601 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20260808 01:51:21.020823 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:21.021973 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:21.028367 7288 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.028456 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.039654 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.317: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.026818 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.047458 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.051295 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.051829 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.054557 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 348: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.056779 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.057520 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 345: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.065541 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.046512 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.076330 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.079391 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.080272 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.080925 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 316: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.081528 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.088464 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.089722 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.092782 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.091506 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.099411 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20260808 01:51:21.100023 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.100386 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.101483 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.001s
I20260808 01:51:21.109417 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.111384 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.114677 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.121559 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.123359 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.125842 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:21.125870 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
W20260808 01:51:21.126219 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.126783 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 316 to 317, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 317 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.127291 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:21.134480 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.139413 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.140338 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 2.319, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.147672 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 3.317, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.149577 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch)
I20260808 01:51:21.152005 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:21.153935 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:21.155531 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 317. Preceding OpId from leader: term: 3 index: 318. (index mismatch)
I20260808 01:51:21.157289 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
W20260808 01:51:21.159379 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.160923 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.162904 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.163699 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.165865 7279 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.171016 7288 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.176833 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.178380 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.179663 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.179876 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.180994 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.186244 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 2.349, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.179898 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.190606 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 349. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260808 01:51:21.192201 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:21.186167 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.205533 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.213205 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.210969 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.215148 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.216092 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 2.324, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.220278 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 324. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
I20260808 01:51:21.220585 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.221797 7288 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:21.222487 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.228688 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 1 index: 323. (index mismatch)
I20260808 01:51:21.228588 7279 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.232270 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.232645 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.235124 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:21.235466 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.238790 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.241353 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 317, Committed index: 317, Last appended: 2.317, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.245558 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20260808 01:51:21.246783 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:21.247006 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
W20260808 01:51:21.247432 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.249780 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 317. Preceding OpId from leader: term: 2 index: 318. (index mismatch)
I20260808 01:51:21.251955 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 318, Last known committed idx: 317, Time since last communication: 0.000s
I20260808 01:51:21.253203 7281 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.259308 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.263942 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.267050 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.318: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.271567 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 317: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.274422 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.287155 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.287549 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.288689 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 349: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.289395 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 346: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.295689 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.299811 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.302575 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.309274 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.310421 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
I20260808 01:51:21.311800 7278 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:21.314476 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20260808 01:51:21.311642 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.316143 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.001s
I20260808 01:51:21.325178 7281 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.337671 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.340867 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.342458 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 2.325, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.342507 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.343214 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.344115 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 317 to 318, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 318 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.348523 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.348897 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.345335 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.353433 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 3.318, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.358637 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 325. Preceding OpId from leader: term: 2 index: 326. (index mismatch)
I20260808 01:51:21.362545 7278 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:21.363730 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.367179 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 318. Preceding OpId from leader: term: 3 index: 319. (index mismatch)
I20260808 01:51:21.369179 7279 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:21.370250 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.367179 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
I20260808 01:51:21.372538 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.380146 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.001s
I20260808 01:51:21.382122 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.384654 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 2.350, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.386377 7044 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.390676 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20260808 01:51:21.392364 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:21.392302 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 2.318, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.393016 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.393395 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.400077 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.402890 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 318. Preceding OpId from leader: term: 2 index: 319. (index mismatch)
I20260808 01:51:21.404819 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20260808 01:51:21.410452 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.405491 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20260808 01:51:21.412042 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.414747 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.416412 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:21.418469 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 1 index: 324. (index mismatch)
I20260808 01:51:21.417331 7294 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.419960 7279 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.001s
I20260808 01:51:21.430049 7238 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.431505 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.443743 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.444169 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.447731 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.432674 7287 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.438186 7284 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.459522 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.463671 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.468150 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.473417 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.463299 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.491153 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 350: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.490253 7279 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.496019 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.498564 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.499289 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.500525 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 347: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.507402 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.508091 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.508447 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20260808 01:51:21.509925 7294 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:21.516285 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.319: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.517143 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.524055 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.528292 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.532074 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.533591 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.534896 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.536592 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.537786 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.541416 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 2.326, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.551985 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.558138 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 326. Preceding OpId from leader: term: 2 index: 327. (index mismatch)
I20260808 01:51:21.559599 7288 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.001s
I20260808 01:51:21.565673 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.567065 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.568094 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.572402 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 318: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.576468 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.577876 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.583012 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 1.324, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.583642 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:21.589344 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 324. Preceding OpId from leader: term: 1 index: 325. (index mismatch)
I20260808 01:51:21.591194 7284 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.001s
I20260808 01:51:21.591697 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.596203 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 2.324, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.603693 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.608101 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 2.319, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.608101 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.617933 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.618314 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.621977 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.622332 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.623063 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.624774 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.628548 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch)
I20260808 01:51:21.627431 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 318 to 319, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.629963 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20260808 01:51:21.632431 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 324. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
I20260808 01:51:21.633787 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:21.636003 7279 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.650080 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:21.652891 7288 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:21.653008 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.654197 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.665819 7288 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.666880 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 2.327, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.667683 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20260808 01:51:21.656836 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.670265 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20260808 01:51:21.671577 7281 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:21.672955 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:21.674088 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.681537 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 3.319, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.681025 7294 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.686064 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 327. Preceding OpId from leader: term: 2 index: 328. (index mismatch)
I20260808 01:51:21.687124 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.695093 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.704840 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:21.705297 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.001s
W20260808 01:51:21.706724 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.707988 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.708329 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.709861 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 319. Preceding OpId from leader: term: 3 index: 320. (index mismatch)
I20260808 01:51:21.712502 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.713390 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:21.732659 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 44158.6us
I20260808 01:51:21.733580 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.735006 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.736860 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.737106 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.738056 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.739362 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.742631 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.744175 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 348: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.745469 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20260808 01:51:21.748191 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 353. (index mismatch)
I20260808 01:51:21.751989 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:21.757217 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.757920 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.770200 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.772039 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.776640 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:21.784879 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.792661 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 1.325, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.799352 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.799747 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.804064 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.814744 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.816123 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.818522 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.824007 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.828596 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20260808 01:51:21.829865 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 325. Preceding OpId from leader: term: 1 index: 326. (index mismatch)
I20260808 01:51:21.830456 7288 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.837967 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 351: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.849907 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:21.850687 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.852738 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:21.855005 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.865417 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.869796 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:21.869717 7238 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.001s
I20260808 01:51:21.872538 7279 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.874140 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20260808 01:51:21.876194 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.878340 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.880954 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:21.882349 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 2.325, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.886152 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20260808 01:51:21.891873 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.892272 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.892443 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:21.893436 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:21.894124 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.900671 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.902942 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.907805 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.908208 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.911681 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 319: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.920446 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.320: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.923673 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
I20260808 01:51:21.925213 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 325. Preceding OpId from leader: term: 2 index: 326. (index mismatch)
I20260808 01:51:21.927579 7288 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.930461 7284 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.931529 7279 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.001s
I20260808 01:51:21.934613 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 349: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.935667 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:21.937458 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.936614 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.948768 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.950932 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.954341 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.962852 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.965245 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:21.975296 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 2.328, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:21.981968 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.985219 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:21.986441 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:21.988737 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:21.996124 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:21.996482 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 328. Preceding OpId from leader: term: 2 index: 329. (index mismatch)
I20260808 01:51:21.999588 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:21.999994 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:21.998581 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.003278 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:22.007494 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.010672 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:22.012709 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20260808 01:51:22.014288 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:22.015679 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.021294 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.007413 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.026952 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 319 to 320, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.028685 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.029903 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.030260 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.032195 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.033226 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.046224 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 352: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.047003 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.048530 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.075537 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 1.354, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.077872 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.082355 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20260808 01:51:22.082757 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 3.320, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.085919 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:22.086959 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 320. Preceding OpId from leader: term: 3 index: 321. (index mismatch)
I20260808 01:51:22.087201 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.087615 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.089520 7278 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20260808 01:51:22.089520 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.091657 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.097745 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.098875 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.108098 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.091209 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20260808 01:51:22.096238 7278 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.321: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.111006 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:22.104902 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.119493 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.121166 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.106280 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 2.326, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.095933 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 1.326, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.129392 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 326. Preceding OpId from leader: term: 2 index: 327. (index mismatch)
I20260808 01:51:22.129858 7281 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.131498 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:22.132203 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 326. Preceding OpId from leader: term: 1 index: 327. (index mismatch)
I20260808 01:51:22.134949 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:22.143558 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.144294 7278 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.145932 7284 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.147616 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.141587 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.155061 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20260808 01:51:22.156836 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.156165 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:22.161525 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:22.162336 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.162886 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.004s
I20260808 01:51:22.158010 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.170080 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20260808 01:51:22.170495 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 2.329, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.171582 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:22.179901 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.182217 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 350: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.188207 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.189929 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 320 to 321, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.190910 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:22.191087 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.190495 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:22.191974 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.192701 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.193126 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.193487 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
I20260808 01:51:22.192746 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.194720 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.194840 7279 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:22.197266 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.192775 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 329. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20260808 01:51:22.200455 7279 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:22.202327 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.206601 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.210920 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 353: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.221235 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.223922 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.229498 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 3.321, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.230484 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.233032 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.242801 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.243839 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.247881 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 1.355, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.252688 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 321. Preceding OpId from leader: term: 3 index: 322. (index mismatch)
I20260808 01:51:22.253916 7278 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260808 01:51:22.254141 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.256237 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.256580 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.257838 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.258594 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20260808 01:51:22.260156 7281 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:22.260517 7278 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.260814 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.261972 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.268079 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.270020 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.271062 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.322: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.278621 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.281531 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 321: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.263027 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.282755 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.283955 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.284623 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.290460 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20260808 01:51:22.302752 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 2.354, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.306463 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:22.309369 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.309769 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.312697 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.319691 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.324004 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 354. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20260808 01:51:22.326591 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:22.330698 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.333240 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.334813 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.337242 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 2.324, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.332410 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.342478 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 321 to 322, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.343007 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.348489 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20260808 01:51:22.350062 7284 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:22.354256 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 354: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.358318 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.361305 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 1.356, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.363065 5071 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.378499 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20260808 01:51:22.380815 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 351: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.381944 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:22.386286 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 324. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
I20260808 01:51:22.387740 7278 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:22.393581 7288 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.395044 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.402989 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.403878 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.411409 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.412817 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.415997 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.419852 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.422421 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.327, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.423815 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 2.355, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.427747 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 355. Preceding OpId from leader: term: 2 index: 356. (index mismatch)
I20260808 01:51:22.429159 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:22.429136 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.430647 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.432147 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.433384 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.435325 7281 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.440117 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.440502 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.447484 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.448316 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 355: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.450623 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20260808 01:51:22.454964 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:22.463582 7284 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.465471 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.467873 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.473747 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.474116 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.479449 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.482286 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 353. (index mismatch)
I20260808 01:51:22.483649 7281 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:22.484856 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.485224 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.486186 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20260808 01:51:22.490598 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:22.491744 7281 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.494465 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.499758 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.509714 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 2.356, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.515569 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.518996 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:22.520706 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.521459 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.532577 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 356. Preceding OpId from leader: term: 2 index: 357. (index mismatch)
W20260808 01:51:22.537281 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.538574 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.539976 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:22.549727 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 352: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.556298 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.558979 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.549419 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.575799 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.577889 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.579169 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.590454 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 356: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.592797 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20260808 01:51:22.596349 7322 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:22.599119 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.603590 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.611892 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 1.358, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.612651 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 2.357, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.609836 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 2.325, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.615119 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 353: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.618214 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20260808 01:51:22.621129 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
I20260808 01:51:22.622612 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:22.622498 7322 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.001s
I20260808 01:51:22.622506 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.625185 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.629704 7322 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.635157 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.639636 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.640674 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.641053 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.644409 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 325. Preceding OpId from leader: term: 2 index: 326. (index mismatch)
I20260808 01:51:22.645040 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.645553 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.645920 7278 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:22.648064 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.655961 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
I20260808 01:51:22.657384 7278 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:22.661834 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 2.327, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.662859 7044 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.665303 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.666349 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.670953 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.671875 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.673046 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.673758 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.677721 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.681329 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.682577 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.687544 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 357: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.692063 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 1.354, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.686086 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 2.330, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.687723 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.697034 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 3.322, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.697520 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 330. Preceding OpId from leader: term: 2 index: 331. (index mismatch)
I20260808 01:51:22.699411 7278 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:22.702805 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.703166 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.706718 5067 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20260808 01:51:22.708793 7278 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:22.709357 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 322. Preceding OpId from leader: term: 3 index: 323. (index mismatch)
I20260808 01:51:22.712646 7288 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.714339 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20260808 01:51:22.715718 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:22.717540 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.719074 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.720472 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 327. Preceding OpId from leader: term: 2 index: 328. (index mismatch)
I20260808 01:51:22.720547 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.721712 7322 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.724346 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.726542 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:22.733765 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.711967 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260808 01:51:22.743644 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.745448 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.743656 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.757295 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.757970 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.761992 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.768895 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 2.326, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.771095 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.773036 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 2.324, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.775718 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.780049 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.781565 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.787009 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.788285 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.804769 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 2.328, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:22.806473 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 51013.5us
I20260808 01:51:22.807353 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 354: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.810074 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 2.331, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.815114 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.323: config changed from index 322 to 323, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.816974 7325 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.821635 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 326. Preceding OpId from leader: term: 2 index: 327. (index mismatch)
I20260808 01:51:22.823710 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 324. Preceding OpId from leader: term: 2 index: 325. (index mismatch)
I20260808 01:51:22.823976 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 3.323, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.826824 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 1.359, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.827508 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:22.829025 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:22.833892 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.834235 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.845886 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.846460 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.854153 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 331. Preceding OpId from leader: term: 2 index: 332. (index mismatch)
I20260808 01:51:22.850809 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.859616 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.001s
I20260808 01:51:22.863201 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 359. Preceding OpId from leader: term: 1 index: 360. (index mismatch)
I20260808 01:51:22.865509 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.001s
I20260808 01:51:22.867739 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.870179 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
I20260808 01:51:22.872354 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:22.872484 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 1.355, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.878513 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.882056 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.884855 7278 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.889878 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:22.889599 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:22.890679 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.891606 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.892091 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.895910 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20260808 01:51:22.897251 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.899165 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.902235 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.904524 7288 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.906001 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 323. Preceding OpId from leader: term: 3 index: 324. (index mismatch)
I20260808 01:51:22.908231 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 328. Preceding OpId from leader: term: 2 index: 329. (index mismatch)
I20260808 01:51:22.910261 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 2.325, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:22.915016 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.914661 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.921224 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:22.922405 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:22.923604 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.925581 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 358: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.926857 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:22.926499 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.928231 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20260808 01:51:22.930125 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:22.933470 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.939685 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.953333 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:22.952873 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.956103 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.964238 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.967154 7325 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:22.970273 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.973887 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 355: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:22.977342 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 323: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.977228 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 2.359, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:22.981928 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:22.982661 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.986841 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.988011 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:22.978192 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:22.979812 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 325. Preceding OpId from leader: term: 2 index: 326. (index mismatch)
I20260808 01:51:22.980803 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.324: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
I20260808 01:51:23.004343 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:51:23.005635 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 329 | No
B | A B* C~ | 2 | 328 | Yes
C | [config not available] | | |
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 1 | 354 | Yes
B | A* B | 1 | 353 | Yes
C | [config not available] | | |
Tablet 24ad2932d9a1489b96a63bc762f6a23f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 326 | Yes
B | A B* C~ | 2 | 326 | No
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:23.014905 7238 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 353 | Yes
B | A* B | 2 | 352 | Yes
C | [config not available] | | |
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 323 | Yes
B | A B* | 2 | 323 | No
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B | 1 | 350 | Yes
B | A* B C~ | 1 | 349 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 4 | 0 | 0 | 6
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 7
Median | 10
Third Quartile | 10
Maximum | 10
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 27
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:23.026777 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.026965 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.020160 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 359. Preceding OpId from leader: term: 2 index: 360. (index mismatch)
I20260808 01:51:23.031745 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:23.022650 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 2.327, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.035799 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:23.025750 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.041601 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 1.356, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.044495 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.044870 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.046175 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.052423 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.052835 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.053251 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 327. Preceding OpId from leader: term: 2 index: 328. (index mismatch)
I20260808 01:51:23.054834 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
I20260808 01:51:23.056646 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20260808 01:51:23.057390 7288 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:23.058481 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.001s
I20260808 01:51:23.059790 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.061275 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.002s
I20260808 01:51:23.061646 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.063238 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.064517 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 323 to 324, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.066103 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.072618 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.074065 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.071645 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.083405 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.088862 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.094059 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 356: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.099360 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 1.360, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.096474 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.092819 7325 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.100238 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.101418 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.104343 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 360. Preceding OpId from leader: term: 1 index: 361. (index mismatch)
I20260808 01:51:23.107733 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 2.326, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.106035 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:23.111961 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 326. Preceding OpId from leader: term: 2 index: 327. (index mismatch)
I20260808 01:51:23.117105 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:23.131196 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.132048 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 359: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.133893 7278 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.134954 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 1.329, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.135352 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.138887 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.141724 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20260808 01:51:23.143195 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:23.150686 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.151106 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.152034 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 3.324, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.152130 7288 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.154367 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.155025 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.145692 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.158959 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 2.329, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.165024 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 324. Preceding OpId from leader: term: 3 index: 325. (index mismatch)
I20260808 01:51:23.167621 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:23.167021 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:23.168560 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.170313 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 329. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20260808 01:51:23.172066 7044 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I20260808 01:51:23.173323 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:23.182731 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.182896 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20260808 01:51:23.184329 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:23.189388 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.191560 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 2.360, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.192914 7325 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.200455 7284 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.202416 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.202808 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.201583 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.206665 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.207624 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.207181 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.211090 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.211741 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.212348 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.325: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.211813 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.214335 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.215488 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.219079 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 324: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.226969 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 360. Preceding OpId from leader: term: 2 index: 361. (index mismatch)
I20260808 01:51:23.228408 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:23.232703 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 357: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.231489 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.239513 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 1.361, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.241590 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.249747 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.250151 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.250952 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 361. Preceding OpId from leader: term: 1 index: 362. (index mismatch)
I20260808 01:51:23.252390 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:23.254387 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.259872 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 360: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.265309 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.266788 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 324 to 325, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 325 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.268616 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 1.330, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.268505 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 2.333, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.264811 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 2.328, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.270401 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.274612 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 328. Preceding OpId from leader: term: 2 index: 329. (index mismatch)
I20260808 01:51:23.274602 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 2 index: 334. (index mismatch)
I20260808 01:51:23.275676 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.276034 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.276607 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:23.277750 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:23.284963 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 330. Preceding OpId from leader: term: 1 index: 331. (index mismatch)
I20260808 01:51:23.283888 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.287514 7288 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:23.289129 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.290233 7325 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.296329 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.308928 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 1.358, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.309646 7278 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.310729 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.315012 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.315425 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.315081 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.316721 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20260808 01:51:23.316447 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.323210 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.001s
I20260808 01:51:23.329282 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.330623 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.330941 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.332371 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.332888 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.335045 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.338209 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.341377 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.352116 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 358: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.356528 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 2.361, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.357743 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.361032 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 361. Preceding OpId from leader: term: 2 index: 362. (index mismatch)
I20260808 01:51:23.363263 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:23.367002 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch)
I20260808 01:51:23.367017 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.368312 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.369773 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:23.370786 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.375255 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.388952 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.390825 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 1.331, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.400983 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 1.359, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.405001 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 2.330, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.407660 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 2.327, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.411291 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 330. Preceding OpId from leader: term: 2 index: 331. (index mismatch)
I20260808 01:51:23.415026 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.415371 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.414717 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.415881 7278 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:23.416721 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 327. Preceding OpId from leader: term: 2 index: 328. (index mismatch)
I20260808 01:51:23.418087 7278 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:23.418310 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 359. Preceding OpId from leader: term: 1 index: 360. (index mismatch)
I20260808 01:51:23.419831 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:23.421326 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.422797 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20260808 01:51:23.423250 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 361: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.425031 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.425408 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.428696 7325 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:23.428418 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.430240 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.430555 7325 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.430048 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 3.325, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.432305 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.433939 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.443023 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 359: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.446002 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:23.447966 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 325. Preceding OpId from leader: term: 3 index: 326. (index mismatch)
I20260808 01:51:23.447248 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:23.449657 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.450675 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20260808 01:51:23.453765 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.456363 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 2.362, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.462019 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 362. Preceding OpId from leader: term: 2 index: 363. (index mismatch)
I20260808 01:51:23.463411 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.463788 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.466282 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.475219 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.475520 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.478843 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.480075 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.487867 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.490962 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.493712 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.495054 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.497766 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:23.499626 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 2.334, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.503137 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.326: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.508916 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 334. Preceding OpId from leader: term: 2 index: 335. (index mismatch)
I20260808 01:51:23.511124 7044 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:23.512171 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 2.329, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.516587 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 1.363, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.517795 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 1.360, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.520103 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.523773 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 360. Preceding OpId from leader: term: 1 index: 361. (index mismatch)
I20260808 01:51:23.525296 7280 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:23.526794 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.530608 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.532580 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 363. Preceding OpId from leader: term: 1 index: 364. (index mismatch)
I20260808 01:51:23.534024 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.534368 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.534742 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 362: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.535756 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:23.539690 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.540503 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.546604 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:23.546088 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.547812 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.547652 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.548715 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 329. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
W20260808 01:51:23.549788 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.549060 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 325 to 326, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.552488 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.556149 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:23.556306 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.556823 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.562992 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.569489 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 360: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.571075 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 325: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.574800 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.579087 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.582420 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.583866 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.585518 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.589473 7325 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.595253 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.595206 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.598167 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.607486 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 2.363, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.608263 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 1.364, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.608596 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 1.361, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.612135 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 363. Preceding OpId from leader: term: 2 index: 364. (index mismatch)
I20260808 01:51:23.612766 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 364. Preceding OpId from leader: term: 1 index: 365. (index mismatch)
I20260808 01:51:23.613425 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.001s
I20260808 01:51:23.614027 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:23.617035 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.332, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.622846 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 2.331, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.622874 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.624185 4781 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.628149 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.629957 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.630327 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.630504 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.632529 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 361. Preceding OpId from leader: term: 1 index: 362. (index mismatch)
I20260808 01:51:23.639997 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:23.642791 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20260808 01:51:23.644351 7044 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:23.639966 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 3.326, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.648927 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.652984 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.651075 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.651600 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.653240 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.642772 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 331. Preceding OpId from leader: term: 2 index: 332. (index mismatch)
I20260808 01:51:23.659268 7325 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:23.660446 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 363: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.669368 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.669016 7284 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.672564 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 326. Preceding OpId from leader: term: 3 index: 327. (index mismatch)
I20260808 01:51:23.673839 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I20260808 01:51:23.677165 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.679299 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.679818 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.686385 7327 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.687965 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.327: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.692065 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 361: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.690008 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.699414 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 326: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.701481 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 2.330, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.702575 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 2.328, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.706182 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 330. Preceding OpId from leader: term: 2 index: 331. (index mismatch)
I20260808 01:51:23.709985 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 328. Preceding OpId from leader: term: 2 index: 329. (index mismatch)
I20260808 01:51:23.704584 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 2.335, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.708145 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:23.712168 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:23.712778 4781 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.714325 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.715677 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.717026 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:23.716811 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:23.717372 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.718909 4781 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.720161 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20260808 01:51:23.720224 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 326 to 327, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.722002 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:23.724334 7327 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.725255 7284 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.731472 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.735759 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.737387 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.739035 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.743644 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.746299 7325 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.750464 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.750826 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.753497 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.755791 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.757187 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch)
I20260808 01:51:23.758131 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.759428 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:23.762220 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
I20260808 01:51:23.763744 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:23.767211 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.767576 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.767143 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.768165 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.769344 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.773170 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20260808 01:51:23.776182 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 362: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.778386 7327 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:23.779215 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.780707 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.792250 7284 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.797255 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 1.365, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.798420 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.802584 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.803296 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.807542 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 2.364, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.812451 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 3.327, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.813252 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 364. Preceding OpId from leader: term: 2 index: 365. (index mismatch)
I20260808 01:51:23.813787 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 365. Preceding OpId from leader: term: 1 index: 366. (index mismatch)
I20260808 01:51:23.814759 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.815140 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.816422 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:23.816094 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.816561 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:23.817593 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.819247 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.819597 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.819999 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.821465 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.823167 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.823589 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.824632 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.825883 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.829025 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.829402 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.831773 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.834154 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 327. Preceding OpId from leader: term: 3 index: 328. (index mismatch)
I20260808 01:51:23.835522 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.837950 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20260808 01:51:23.842458 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 364: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.845300 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 1.363, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.848979 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.849709 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.851995 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 363. Preceding OpId from leader: term: 1 index: 364. (index mismatch)
I20260808 01:51:23.854068 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.001s
I20260808 01:51:23.855795 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 2.331, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.856494 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.857851 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 2.333, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.855830 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.328: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.861897 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 331. Preceding OpId from leader: term: 2 index: 332. (index mismatch)
I20260808 01:51:23.862130 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.862466 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.863338 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:23.858821 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.867267 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.867621 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.866663 7280 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.868496 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 2 index: 334. (index mismatch)
I20260808 01:51:23.870009 7044 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:23.869966 7327 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.876121 7325 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.877583 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.879789 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.882016 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.883395 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.885977 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.888557 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 327: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.889637 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 363: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.893110 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.893342 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.898813 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20260808 01:51:23.902595 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.903527 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 2.365, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.903430 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 2.329, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.905063 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.906634 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20260808 01:51:23.907210 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.908036 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:23.908567 7327 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:23.912950 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.913317 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.913491 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20260808 01:51:23.914754 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 365. Preceding OpId from leader: term: 2 index: 366. (index mismatch)
I20260808 01:51:23.915395 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 329. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20260808 01:51:23.914211 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.916265 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:23.917059 7327 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.918057 7044 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20260808 01:51:23.921037 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 327 to 328, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.922149 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.928596 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.930843 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:23.931377 7288 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.939383 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.939350 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:23.941601 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.942673 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.943490 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.947129 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.949487 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:23.950146 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.950330 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.955873 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.957873 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 1.364, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.963111 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 364. Preceding OpId from leader: term: 1 index: 365. (index mismatch)
I20260808 01:51:23.965413 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 365: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:23.966770 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:23.966301 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.970093 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.973544 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 3.328, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:23.973869 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
I20260808 01:51:23.975204 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:23.975227 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 2.334, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:23.976864 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.977346 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:23.979601 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 328. Preceding OpId from leader: term: 3 index: 329. (index mismatch)
I20260808 01:51:23.982122 7327 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20260808 01:51:23.984416 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:23.987509 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 1.367, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:23.988018 7044 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.990516 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 334. Preceding OpId from leader: term: 2 index: 335. (index mismatch)
I20260808 01:51:23.991866 7288 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:23.992465 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 367. Preceding OpId from leader: term: 1 index: 368. (index mismatch)
I20260808 01:51:23.993713 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:23.994051 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:23.995157 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:23.990999 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:23.992970 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.329: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.997601 7284 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.998476 7278 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:23.999182 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.000736 5068 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.003149 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.006799 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.009985 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.012127 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.013574 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.014689 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.021652 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.026454 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.029697 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.030824 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.036232 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.036345 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch)
W20260808 01:51:24.036631 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.038067 7327 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:24.040012 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.040827 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 364: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:51:24.045518 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 328: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 328 but the committed config has opid_index of 329
I20260808 01:51:24.046942 7044 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.048614 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.050761 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 2.337, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.047686 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 2.330, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.057209 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 2.366, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.060267 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.063380 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.064451 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 337. Preceding OpId from leader: term: 2 index: 338. (index mismatch)
I20260808 01:51:24.064272 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.065793 7284 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:24.065474 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.067757 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.068231 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.068364 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.067260 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 366. Preceding OpId from leader: term: 2 index: 367. (index mismatch)
I20260808 01:51:24.069962 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 330. Preceding OpId from leader: term: 2 index: 331. (index mismatch)
I20260808 01:51:24.069505 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.070626 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:24.071468 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:24.070896 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 328 to 329, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.076553 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.078150 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.081210 7284 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.081961 7325 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.089063 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 366: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.089599 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.094501 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.095152 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.096608 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.103804 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.104017 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.117537 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.122051 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 1.365, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.122555 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 2.335, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.128185 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.128875 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 2.333, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.129658 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.131369 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.133206 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20260808 01:51:24.133644 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 3.329, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.133206 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 365. Preceding OpId from leader: term: 1 index: 366. (index mismatch)
I20260808 01:51:24.135669 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20260808 01:51:24.137215 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.001s
I20260808 01:51:24.137986 7284 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:24.138973 7280 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:24.143249 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 329. Preceding OpId from leader: term: 3 index: 330. (index mismatch)
I20260808 01:51:24.144768 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.145157 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.144749 7284 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.001s
I20260808 01:51:24.149840 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.150195 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.148048 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 1.336, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.154417 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 2 index: 334. (index mismatch)
I20260808 01:51:24.155006 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 336. Preceding OpId from leader: term: 1 index: 337. (index mismatch)
I20260808 01:51:24.155682 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.156834 7325 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:24.158514 7325 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:24.165809 7278 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.169973 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.170394 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.171803 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 2.338, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.174365 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.157748 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.179482 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 2.331, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.172425 7327 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.186050 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.187322 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.187330 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 338. Preceding OpId from leader: term: 2 index: 339. (index mismatch)
I20260808 01:51:24.192482 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:24.193123 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.194478 7288 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.203697 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.204281 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 365: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:24.205739 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.209286 5068 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.330: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.209859 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.213516 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.214898 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 329: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.219924 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 368: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.220886 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.222800 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 331. Preceding OpId from leader: term: 2 index: 332. (index mismatch)
I20260808 01:51:24.223121 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.217131 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 2.367, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.226322 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:24.225510 7336 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.230734 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.231698 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 367. Preceding OpId from leader: term: 2 index: 368. (index mismatch)
I20260808 01:51:24.233009 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:24.234097 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.238687 7327 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.244522 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.228909 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.250133 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.252136 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.253573 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 329 to 330, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.254379 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.254966 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.256007 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.260726 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.271639 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.274926 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 1.337, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.277673 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20260808 01:51:24.279227 7327 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:24.283790 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.283684 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 367: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:24.284135 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.285071 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 337. Preceding OpId from leader: term: 1 index: 338. (index mismatch)
I20260808 01:51:24.286505 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:24.286435 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 3.330, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.292389 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.294755 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.296921 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 2.334, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.302042 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.303661 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.290438 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 330. Preceding OpId from leader: term: 3 index: 331. (index mismatch)
I20260808 01:51:24.305202 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.307404 7288 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I20260808 01:51:24.307993 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.312076 5068 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 334. Preceding OpId from leader: term: 2 index: 335. (index mismatch)
I20260808 01:51:24.311728 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.314527 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.322476 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20260808 01:51:24.326025 7325 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.329198 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.333465 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.331: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.336383 7336 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.337937 5067 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.350837 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 330: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.353149 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:24.368454 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 334 but the committed config has opid_index of 335
I20260808 01:51:24.370682 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 2.332, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.366436 4781 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.373757 4781 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.379891 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 332. Preceding OpId from leader: term: 2 index: 333. (index mismatch)
I20260808 01:51:24.379575 4781 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.381232 7336 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20260808 01:51:24.381448 4781 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.390476 7325 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.402297 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.407828 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.409181 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 330 to 331, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 331 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.410805 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.424834 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 2.337, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.431854 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 1.338, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.439179 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.439579 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.440979 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 2.335, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.444698 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 3.331, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.446398 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20260808 01:51:24.448726 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.454521 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.452775 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 331. Preceding OpId from leader: term: 3 index: 332. (index mismatch)
I20260808 01:51:24.459105 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.453730 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.460146 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 2.368, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.460848 7288 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:24.462193 7288 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20260808 01:51:24.464926 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20260808 01:51:24.468768 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.470950 7336 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.471529 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 2.339, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.471323 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:24.465873 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.475790 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.476147 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:24.476509 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.477187 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.477514 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.478250 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:24.480384 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch)
I20260808 01:51:24.480350 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.490036 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.490501 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.491784 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 368. Preceding OpId from leader: term: 2 index: 369. (index mismatch)
I20260808 01:51:24.493753 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.486353 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.481220 5067 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.332: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.482105 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 337. Preceding OpId from leader: term: 2 index: 338. (index mismatch)
I20260808 01:51:24.509922 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:24.492240 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.496644 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 338. Preceding OpId from leader: term: 1 index: 339. (index mismatch)
I20260808 01:51:24.518114 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.519908 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 2.333, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.502081 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 339. Preceding OpId from leader: term: 2 index: 340. (index mismatch)
I20260808 01:51:24.523131 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.523650 7288 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:24.524327 7327 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:24.496186 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.030s
I20260808 01:51:24.525309 7288 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.001s
I20260808 01:51:24.532327 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.534157 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 2 index: 334. (index mismatch)
I20260808 01:51:24.536906 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.537340 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.537976 7288 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20260808 01:51:24.539752 7288 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.541671 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.544514 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.546949 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:24.554095 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:24.554575 5010 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.68:35651, user_credentials={real_user=slave}} blocked reactor thread for 68353us
I20260808 01:51:24.564116 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 369: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.573289 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 368: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.578696 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 366: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.581938 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.585992 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.586506 7327 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.594673 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.594314 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.595377 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 331: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.596028 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.620618 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.622813 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.594848 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.626596 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
W20260808 01:51:24.647207 5146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.66:43479, user_credentials={real_user=slave}} blocked reactor thread for 63287.1us
I20260808 01:51:24.652025 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.652523 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.661141 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:24.673386 5067 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 1.370, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.681360 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.682945 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.684388 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.688146 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 371. (index mismatch)
I20260808 01:51:24.689716 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.001s
I20260808 01:51:24.697034 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 1.367, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.699579 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.701447 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 2.369, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.703737 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.711941 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.712858 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20260808 01:51:24.714368 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:24.715548 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 2.340, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.719537 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 2.334, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.722898 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.727231 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.733163 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20260808 01:51:24.733206 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 340. Preceding OpId from leader: term: 2 index: 341. (index mismatch)
I20260808 01:51:24.734628 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.001s
I20260808 01:51:24.735658 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:24.736501 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 370: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.742924 7044 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.750702 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.755815 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 369: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.760555 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.775306 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 367. Preceding OpId from leader: term: 1 index: 368. (index mismatch)
I20260808 01:51:24.779264 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 334. Preceding OpId from leader: term: 2 index: 335. (index mismatch)
I20260808 01:51:24.783329 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.785856 7280 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.001s
I20260808 01:51:24.790994 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.804430 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.806851 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.819311 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.820328 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.821201 7337 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.001s
I20260808 01:51:24.785406 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 331 to 332, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.827824 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 367: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.831317 7337 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.823486 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.835361 5067 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.835428 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:24.859503 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.860836 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.862365 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.863806 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:24.884222 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 340 but the committed config has opid_index of 341
I20260808 01:51:24.884940 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:24.893678 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 2.338, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.894495 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 3.332, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:24.900573 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.912178 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 332. Preceding OpId from leader: term: 3 index: 333. (index mismatch)
I20260808 01:51:24.912737 4781 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:24.915486 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.001s
I20260808 01:51:24.916986 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.917506 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 339. Preceding OpId from leader: term: 1 index: 340. (index mismatch)
W20260808 01:51:24.917506 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.918642 5067 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 338. Preceding OpId from leader: term: 2 index: 339. (index mismatch)
I20260808 01:51:24.919755 7340 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.001s
I20260808 01:51:24.924418 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 1.371, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.930464 7340 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:24.935458 5067 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.941601 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.942014 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.929569 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 2.370, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.952266 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.952699 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.941831 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 2.335, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.945009 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 2.337, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.964377 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 371. Preceding OpId from leader: term: 1 index: 372. (index mismatch)
I20260808 01:51:24.946575 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20260808 01:51:24.968454 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.949640 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 2.341, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:24.970713 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.951165 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 370. Preceding OpId from leader: term: 2 index: 371. (index mismatch)
I20260808 01:51:24.954802 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:24.974239 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.979120 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:24.979502 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.980165 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:24.976033 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 335. Preceding OpId from leader: term: 2 index: 336. (index mismatch)
I20260808 01:51:24.977669 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
W20260808 01:51:24.981297 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.977638 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 337. Preceding OpId from leader: term: 2 index: 338. (index mismatch)
W20260808 01:51:24.981997 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:24.980235 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.333: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.982681 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260808 01:51:24.982941 5067 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.958196 7337 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.985772 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20260808 01:51:24.986277 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20260808 01:51:24.991900 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:24.994297 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.995841 7280 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:24.999725 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.000001 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.007400 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.009198 7338 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.001s
I20260808 01:51:25.009783 7340 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:25.011276 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:25.015204 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 370: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.013501 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.018631 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 371: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.022873 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 332: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.028348 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.031489 7338 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.032656 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.036404 4781 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 332 to 333, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.038767 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.046577 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 368: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.049296 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.029388 5067 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.055516 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.056461 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.059248 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 3.333, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.059878 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.057914 7340 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.067761 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20260808 01:51:25.070786 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20260808 01:51:25.075973 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.083804 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.085110 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.085506 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.087141 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 333. Preceding OpId from leader: term: 3 index: 334. (index mismatch)
I20260808 01:51:25.090000 7278 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.001s
I20260808 01:51:25.090102 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.079504 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.093137 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.091346 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.097739 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.099272 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.096764 7278 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.334: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.103672 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.116122 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.116812 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 333: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.132164 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.139420 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20260808 01:51:25.140241 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.141491 7340 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:25.148072 5067 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 2.371, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.145481 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.152109 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.152650 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.157989 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch)
I20260808 01:51:25.158172 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 371. Preceding OpId from leader: term: 2 index: 372. (index mismatch)
I20260808 01:51:25.159744 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:25.160046 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260808 01:51:25.152842 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 372: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.162360 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.166189 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.166960 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.170063 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.163506 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 333 to 334, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.175755 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 371: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.177142 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.181978 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.185189 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.199045 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.193104 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 369: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.226874 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:25.255968 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 2.338, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.262943 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 338. Preceding OpId from leader: term: 2 index: 339. (index mismatch)
I20260808 01:51:25.264815 7340 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:25.271347 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 2.339, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.272181 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:25.276010 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.277455 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.283977 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.281462 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 339. Preceding OpId from leader: term: 2 index: 340. (index mismatch)
I20260808 01:51:25.298714 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:25.304325 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 3.334, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.307278 7339 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.308907 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.309942 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.311322 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.312777 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.313961 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.316814 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:25.318527 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 2.336, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.321771 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.323640 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20260808 01:51:25.325229 7340 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.001s
I20260808 01:51:25.332064 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 336. Preceding OpId from leader: term: 2 index: 337. (index mismatch)
I20260808 01:51:25.336362 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 334. Preceding OpId from leader: term: 3 index: 335. (index mismatch)
I20260808 01:51:25.338595 7339 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.002s
I20260808 01:51:25.338690 7340 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.006s
I20260808 01:51:25.329314 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.346868 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.347592 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.351125 7339 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.354127 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.363961 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.365406 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.366573 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch)
I20260808 01:51:25.370153 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:25.372615 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:25.372550 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.374096 7238 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.379547 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.402815 5068 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 1.373, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.403802 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:25.403939 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.405293 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:25.406132 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.410084 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 2.340, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.406950 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.411177 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.411867 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 334: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.419131 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.431938 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.412046 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.335: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.442716 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 340. Preceding OpId from leader: term: 2 index: 341. (index mismatch)
I20260808 01:51:25.453169 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 2.339, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.454677 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.455747 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20260808 01:51:25.456261 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 2.337, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.455917 5072 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.457983 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.459648 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.469520 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20260808 01:51:25.473820 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 1.370, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.474677 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:25.482522 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 371. (index mismatch)
I20260808 01:51:25.484392 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20260808 01:51:25.487406 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.487767 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:25.493767 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.496498 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:25.495860 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.498078 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 2.372, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.499229 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.504910 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.505280 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:25.505818 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.507750 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.514739 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.522454 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 372. Preceding OpId from leader: term: 2 index: 373. (index mismatch)
I20260808 01:51:25.524377 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20260808 01:51:25.527047 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:25.537899 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 339. Preceding OpId from leader: term: 2 index: 340. (index mismatch)
I20260808 01:51:25.542832 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 337. Preceding OpId from leader: term: 2 index: 338. (index mismatch)
I20260808 01:51:25.559521 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:25.561739 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:25.567490 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.580948 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 372: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.588668 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.600961 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.610172 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.618606 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.622727 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.632864 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.636812 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 370: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.639230 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.596772 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.644186 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 334 to 335, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.650730 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.646242 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 373: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.662051 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.665876 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.672222 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.673836 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.686895 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 1.371, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.691902 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 371. Preceding OpId from leader: term: 1 index: 372. (index mismatch)
I20260808 01:51:25.693567 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260808 01:51:25.700457 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.703172 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.722689 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 1.342, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.728065 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch)
I20260808 01:51:25.729030 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 1.374, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.729751 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:25.733762 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20260808 01:51:25.736238 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 3.335, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.739060 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.739946 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260808 01:51:25.744006 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 2.343, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.735982 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.748718 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.751935 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.756166 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.756532 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.756759 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 343. Preceding OpId from leader: term: 2 index: 344. (index mismatch)
I20260808 01:51:25.757529 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 335. Preceding OpId from leader: term: 3 index: 336. (index mismatch)
I20260808 01:51:25.758487 7338 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:25.760391 7338 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20260808 01:51:25.765488 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 371: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.772819 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.747350 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.778280 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.780128 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.781270 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.781186 7339 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.782708 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:25.784220 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.790021 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.336: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.787407 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.822244 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.826448 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.829766 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 374: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.839221 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:25.846168 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.848506 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 335: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.867986 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 2.338, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.871248 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 2.340, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.879624 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 340. Preceding OpId from leader: term: 2 index: 341. (index mismatch)
I20260808 01:51:25.880483 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 1.375, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.880810 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 2.373, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.880836 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.881152 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:25.882606 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 2.341, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.885232 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 373. Preceding OpId from leader: term: 2 index: 374. (index mismatch)
I20260808 01:51:25.886407 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.891261 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.892107 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 338. Preceding OpId from leader: term: 2 index: 339. (index mismatch)
I20260808 01:51:25.892802 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20260808 01:51:25.893893 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.894291 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.894950 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.895296 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.896744 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20260808 01:51:25.898093 7338 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:25.898881 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.899755 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:25.900141 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.900238 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 335 to 336, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.901177 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20260808 01:51:25.903469 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:25.904593 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 375. Preceding OpId from leader: term: 1 index: 376. (index mismatch)
I20260808 01:51:25.904007 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.906167 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s
I20260808 01:51:25.907552 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.909145 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.913801 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.921412 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.928347 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.935199 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20260808 01:51:25.937481 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.946482 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:25.948221 7338 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.951512 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.956138 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:25.959169 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20260808 01:51:25.960124 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 3.336, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:25.960397 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.964394 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:25.964957 7238 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:25.967631 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:25.969141 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:25.972530 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20260808 01:51:25.974856 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:25.974941 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.976130 7340 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:25.979071 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:25.984099 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 375: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:25.972530 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 336. Preceding OpId from leader: term: 3 index: 337. (index mismatch)
I20260808 01:51:25.994874 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.004894 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 372: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.004328 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.010484 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.012315 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.014159 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.015249 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.016649 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.022007 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.029991 7339 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I20260808 01:51:26.032960 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.034679 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 1.376, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.044236 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 1.373, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.047708 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 373: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.050441 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.056471 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.337: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.062671 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.084582 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 336: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.085718 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 376. Preceding OpId from leader: term: 1 index: 377. (index mismatch)
I20260808 01:51:26.087167 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20260808 01:51:26.090492 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I20260808 01:51:26.091863 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20260808 01:51:26.092578 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.093969 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.095218 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.100404 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.112792 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.124634 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 373: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.125433 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.127060 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.134974 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 376: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.136488 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.138163 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 2.341, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.142558 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20260808 01:51:26.145833 7338 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:26.147701 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20260808 01:51:26.149241 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:26.151953 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.152294 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.155966 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 2.339, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.157577 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.159094 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.160604 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.161780 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.162225 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
I20260808 01:51:26.163573 7340 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 1 | 374 | No
B | A* B | 1 | 373 | Yes
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:26.173689 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 339. Preceding OpId from leader: term: 2 index: 340. (index mismatch)
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:26.173926 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.174523 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 2.374, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 3 | 335 | No
B | A B* | 3 | 335 | No
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
I20260808 01:51:26.177732 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 336 to 337, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* | 1 | 341 | Yes
B | A B* C~ | 1 | 342 | No
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 7 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 3
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 23
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:26.183669 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.185635 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.186043 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.192046 7365 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:26.194576 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:26.195545 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 374. Preceding OpId from leader: term: 2 index: 375. (index mismatch)
I20260808 01:51:26.196528 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.198966 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260808 01:51:26.201560 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.203750 7365 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.208196 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.212699 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 1.344, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.220410 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 2.345, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.220737 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 3.337, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.228312 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.228739 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.228312 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.231720 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20260808 01:51:26.231055 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.235507 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:26.237375 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.238699 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 337. Preceding OpId from leader: term: 3 index: 338. (index mismatch)
I20260808 01:51:26.239959 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.240600 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.240612 7339 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I20260808 01:51:26.242527 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.243268 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.240067 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.246064 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 1.374, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.249287 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.254293 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.254665 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.255942 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 1.377, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.256451 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20260808 01:51:26.258318 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260808 01:51:26.262816 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.263185 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.269490 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.275602 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.338: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.279433 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 377. Preceding OpId from leader: term: 1 index: 378. (index mismatch)
I20260808 01:51:26.279774 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 374: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.281536 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20260808 01:51:26.283656 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.291086 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.299268 7338 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.301190 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.302881 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 337: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.305190 7280 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.311906 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.314941 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.323467 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 2.343, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.330029 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 343. Preceding OpId from leader: term: 2 index: 344. (index mismatch)
I20260808 01:51:26.331473 7338 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.002s
I20260808 01:51:26.335045 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.337124 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.336949 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 377: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.338972 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 374: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.339967 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.340286 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.342914 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 344. Preceding OpId from leader: term: 1 index: 345. (index mismatch)
I20260808 01:51:26.340364 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.346618 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 2.340, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.343930 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.344717 5067 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20260808 01:51:26.350481 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 340. Preceding OpId from leader: term: 2 index: 341. (index mismatch)
I20260808 01:51:26.351775 7238 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:26.352967 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:26.359226 7340 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.364897 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.346467 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.367966 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.369716 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 337 to 338, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 338 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.371191 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.372913 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.373832 7339 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:26.374317 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.383281 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.392117 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.392131 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.400494 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.407207 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.408535 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:26.419582 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 340 but the committed config has opid_index of 341
I20260808 01:51:26.421042 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 1.378, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.422739 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.424144 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 2.375, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.428534 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 375. Preceding OpId from leader: term: 2 index: 376. (index mismatch)
I20260808 01:51:26.429550 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20260808 01:51:26.431196 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s
I20260808 01:51:26.432479 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 1.375, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.438359 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 375. Preceding OpId from leader: term: 1 index: 376. (index mismatch)
I20260808 01:51:26.438777 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.440261 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.440853 7280 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s
I20260808 01:51:26.441453 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.441823 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.442929 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 379. (index mismatch)
I20260808 01:51:26.443502 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.444934 7280 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20260808 01:51:26.444989 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.448642 7338 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:26.448348 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.456781 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.457778 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 3.338, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.462028 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.465867 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 375: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.470810 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 338. Preceding OpId from leader: term: 3 index: 339. (index mismatch)
I20260808 01:51:26.472831 7340 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I20260808 01:51:26.474130 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 2.341, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.475498 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.482245 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.482913 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 375: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.483474 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.474931 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.484632 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.485867 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.486354 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.489367 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20260808 01:51:26.488824 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.339: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.490706 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 378: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.491221 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.491464 7338 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:26.496398 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.496915 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.497546 5067 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 341. Preceding OpId from leader: term: 2 index: 342. (index mismatch)
I20260808 01:51:26.496356 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 2.343, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.499913 7338 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.001s
I20260808 01:51:26.501335 7339 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.503400 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.512320 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.518323 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 338: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.518932 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.482354 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:26.521988 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.524150 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 338 to 339, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 339 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.529546 7365 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.533252 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.544801 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 3.339, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:26.548074 5007 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.67:46093, user_credentials={real_user=slave}} blocked reactor thread for 72814.4us
I20260808 01:51:26.549877 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.551203 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 1.376, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.552327 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.552685 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.555567 5067 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20260808 01:51:26.556931 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 339. Preceding OpId from leader: term: 3 index: 340. (index mismatch)
I20260808 01:51:26.559558 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.559957 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.560619 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.564339 7238 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:26.566335 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20260808 01:51:26.572079 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.572342 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 376. Preceding OpId from leader: term: 1 index: 377. (index mismatch)
W20260808 01:51:26.572463 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.573662 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I20260808 01:51:26.549028 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:26.579850 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.580960 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20260808 01:51:26.580642 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.583082 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.582520 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.584523 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.585937 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.591503 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 343. Preceding OpId from leader: term: 2 index: 344. (index mismatch)
I20260808 01:51:26.593307 7367 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.001s
I20260808 01:51:26.594213 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.594343 7238 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.596633 7365 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:26.597460 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.602540 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 376: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.606774 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.608989 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 339: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.610318 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.611297 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.340: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.612178 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.613355 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 2.345, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.619534 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20260808 01:51:26.620573 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.620930 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.623065 7338 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:26.624467 7338 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.626487 7365 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.635357 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.379, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.635834 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 2.376, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.635870 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 339 to 340, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.638388 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.640239 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 2.342, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.642608 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20260808 01:51:26.642863 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.643208 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.644017 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20260808 01:51:26.650589 7326 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.653342 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.653761 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.655087 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.655375 5205 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.659891 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.662526 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 342. Preceding OpId from leader: term: 2 index: 343. (index mismatch)
I20260808 01:51:26.665102 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 379: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.666313 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 376. Preceding OpId from leader: term: 2 index: 377. (index mismatch)
I20260808 01:51:26.668623 7280 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I20260808 01:51:26.670972 7365 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:26.670922 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 1.377, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.673306 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.675282 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.676806 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20260808 01:51:26.677695 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.678357 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.677860 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.680109 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.682670 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.689515 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.689518 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 377. Preceding OpId from leader: term: 1 index: 378. (index mismatch)
I20260808 01:51:26.691635 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.692395 5212 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 3.340, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.694890 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20260808 01:51:26.698331 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch)
I20260808 01:51:26.698336 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.698336 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.701779 7365 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:26.702385 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20260808 01:51:26.702618 7238 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.704975 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.708158 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.710927 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.715273 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.718287 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.720233 5067 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 340. Preceding OpId from leader: term: 3 index: 341. (index mismatch)
I20260808 01:51:26.720651 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.723250 7365 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20260808 01:51:26.724995 7368 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.726572 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.727324 5067 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.732761 5205 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.734226 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.735839 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 376: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.736438 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.740274 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.743377 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.750820 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 340: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.752384 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.341: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.754169 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 380: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.761528 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 2.377, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.766215 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20260808 01:51:26.769042 7365 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:26.770889 5205 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 377. Preceding OpId from leader: term: 2 index: 378. (index mismatch)
I20260808 01:51:26.775106 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 377: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.777395 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20260808 01:51:26.774271 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.777738 7234 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20260808 01:51:26.778997 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.781210 7338 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:26.786934 5068 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.790795 7365 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.792598 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.794518 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 1.381, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.797343 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.798609 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.799955 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.800906 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.378: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.809787 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 377: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.814056 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.814469 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.818080 5205 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 381. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
I20260808 01:51:26.819597 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.820427 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.001s
I20260808 01:51:26.821759 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.823179 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.824450 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.826807 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.828383 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.826208 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 340 to 341, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.838446 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 381: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.843493 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.851711 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.852703 5205 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 2.343, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.857085 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20260808 01:51:26.858554 7338 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:26.862022 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.862385 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.862421 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 343. Preceding OpId from leader: term: 2 index: 344. (index mismatch)
I20260808 01:51:26.863859 7238 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:26.867154 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 1.378, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.867316 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 2.349, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.870867 7238 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.873096 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.874291 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.874661 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.875057 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 379. (index mismatch)
I20260808 01:51:26.876845 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20260808 01:51:26.877918 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.879910 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:26.880290 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.879781 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.881537 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 349. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260808 01:51:26.883572 7365 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.885346 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:26.885819 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.891631 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.893386 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.896083 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 377 to 378, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:26.898517 7367 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.900365 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 378: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.906801 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 3.341, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.909296 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.910904 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.908425 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.913810 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.916325 5068 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 2.378, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.918821 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 1 index: 383. (index mismatch)
I20260808 01:51:26.924188 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.923724 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
W20260808 01:51:26.924569 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.920080 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.005s
I20260808 01:51:26.926501 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.928874 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 378. Preceding OpId from leader: term: 2 index: 379. (index mismatch)
I20260808 01:51:26.931066 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I20260808 01:51:26.933151 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.935379 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.929543 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.946414 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 341. Preceding OpId from leader: term: 3 index: 342. (index mismatch)
I20260808 01:51:26.947537 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.944486 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:26.950489 7234 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.955080 5205 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.379: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:26.960331 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.960575 7238 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20260808 01:51:26.962466 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 378: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:26.965416 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 382: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:26.965899 5212 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:26.974387 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.981196 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20260808 01:51:26.983283 7338 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:26.985090 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:26.986294 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.379, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.986488 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 2.350, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:26.986541 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 2.345, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:26.989549 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
W20260808 01:51:26.985455 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:26.994333 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20260808 01:51:26.993786 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:26.999437 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20260808 01:51:26.999514 7365 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:26.999929 7368 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.001348 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20260808 01:51:27.001289 7365 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:27.003270 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.003645 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.007728 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20260808 01:51:27.011438 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20260808 01:51:27.011389 7367 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:27.014242 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.342: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.015451 7339 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.015795 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.028160 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.032019 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.032248 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 378 to 379, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 379 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.034611 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.035142 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.043939 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.045001 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.046828 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.047137 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.056914 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.060374 7367 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.062668 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 379: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.064590 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 2.379, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.066234 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:27.071204 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 350 but the committed config has opid_index of 351
I20260808 01:51:27.078476 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 379. Preceding OpId from leader: term: 2 index: 380. (index mismatch)
I20260808 01:51:27.079818 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 379, Time since last communication: 0.000s
I20260808 01:51:27.075750 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.084216 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.087160 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.088850 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.091274 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 341: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.092247 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.091641 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 341 to 342, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.093426 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:27.093050 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:27.093824 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.069353 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.097555 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.101441 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20260808 01:51:27.103170 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:27.106890 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.107278 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.107946 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20260808 01:51:27.111153 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 379: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.111024 7367 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.112124 7234 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20260808 01:51:27.113909 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.119172 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.120754 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.123275 7234 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.123984 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20260808 01:51:27.124019 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.126044 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:27.130412 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.130798 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.130615 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.132401 7368 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.135885 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.139094 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch)
I20260808 01:51:27.140532 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20260808 01:51:27.142840 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.146906 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.148576 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 383: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.148480 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.380: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.156540 5205 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.159281 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.161798 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 380: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.166553 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.171521 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.171890 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.171669 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.173153 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20260808 01:51:27.174726 7367 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:27.173106 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.175959 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20260808 01:51:27.176676 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.178046 7367 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:27.178977 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.183650 7238 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.184122 7338 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.186019 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.193068 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.193823 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.194512 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.192778 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.199059 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 379 to 380, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.212309 5205 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.216914 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.218830 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.219209 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.224844 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 1.381, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.227074 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 1.384, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.230485 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 2.380, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.232210 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 2.345, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.235996 5205 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 381. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
I20260808 01:51:27.237394 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20260808 01:51:27.236925 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.240118 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.240506 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.241367 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 380. Preceding OpId from leader: term: 2 index: 381. (index mismatch)
I20260808 01:51:27.241820 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20260808 01:51:27.242063 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 3.342, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.240293 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20260808 01:51:27.250273 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20260808 01:51:27.253684 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20260808 01:51:27.239049 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.257495 7339 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:27.259517 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.259913 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.262233 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20260808 01:51:27.263264 6972 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.266285 7238 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:27.266657 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.268492 7238 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.272119 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.277218 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.277947 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 342. Preceding OpId from leader: term: 3 index: 343. (index mismatch)
I20260808 01:51:27.278205 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.287248 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:27.287801 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.288519 7338 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s
I20260808 01:51:27.291481 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20260808 01:51:27.293016 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:27.296201 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.278857 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.298822 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.299269 5205 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.304721 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 345. Preceding OpId from leader: term: 2 index: 346. (index mismatch)
I20260808 01:51:27.306136 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:27.308049 7339 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20260808 01:51:27.308079 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.315629 5212 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.381: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.322824 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 384: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.327514 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.331933 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 380: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.332247 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20260808 01:51:27.334570 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 381: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.335911 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.336717 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.346536 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.354226 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.354974 7368 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.361236 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.361670 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.365526 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.001s
I20260808 01:51:27.370704 7367 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.379033 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20260808 01:51:27.380707 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20260808 01:51:27.388664 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.395469 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.406121 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.413985 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.415963 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 342: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.414975 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.421221 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.425565 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 385: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.432893 7378 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.441287 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.343: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.443241 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.444756 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.444672 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 380 to 381, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.447105 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.448868 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.450080 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.452780 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.473834 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.476097 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 342 to 343, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.484946 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.485056 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 2.381, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.485857 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.487857 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.489221 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20260808 01:51:27.491868 5212 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 381. Preceding OpId from leader: term: 2 index: 382. (index mismatch)
I20260808 01:51:27.492702 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.493073 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.493276 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20260808 01:51:27.493667 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20260808 01:51:27.498246 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20260808 01:51:27.498986 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 1 index: 383. (index mismatch)
I20260808 01:51:27.500824 7379 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:27.512408 7234 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20260808 01:51:27.513857 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.513923 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.515321 7367 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.523365 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.521377 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.526430 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.382: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.527741 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.529760 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.531109 5205 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 2.349, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.533581 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.534789 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 2.346, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.539119 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 3.343, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.543962 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.544354 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.547274 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 349. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260808 01:51:27.547544 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20260808 01:51:27.547863 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:27.547873 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 346. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
W20260808 01:51:27.548233 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.550387 7339 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:27.550204 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.551123 7367 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.001s
I20260808 01:51:27.553014 7339 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:27.554421 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 381: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.559755 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 343. Preceding OpId from leader: term: 3 index: 344. (index mismatch)
I20260808 01:51:27.563285 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.565163 7339 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20260808 01:51:27.569018 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.572707 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.570322 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.580009 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.582926 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.585801 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 1.351, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.588289 5206 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.590117 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 386: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.585860 7367 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.593649 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 382: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.594599 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.594941 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.598891 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.599264 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.601773 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.605492 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.606227 5068 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.611668 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.611042 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.621788 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.623111 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.623996 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20260808 01:51:27.625226 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 351. Preceding OpId from leader: term: 1 index: 352. (index mismatch)
I20260808 01:51:27.627549 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.634603 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.636129 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 381 to 382, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.639611 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 343: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.642285 7367 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:27.644994 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:27.650319 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.650885 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.344: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.653293 5212 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 2.350, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.659126 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.664759 5212 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20260808 01:51:27.665298 7367 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.662717 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 2.382, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.667522 5212 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 2.349, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.663342 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.670781 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.671409 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20260808 01:51:27.673584 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.673954 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.676924 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 382. Preceding OpId from leader: term: 2 index: 383. (index mismatch)
I20260808 01:51:27.679162 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20260808 01:51:27.679759 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.679193 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.002s
I20260808 01:51:27.680615 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.685045 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.685405 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.685596 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.691279 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.690852 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.692611 5073 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 349. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260808 01:51:27.693882 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:27.695195 5212 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20260808 01:51:27.696595 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20260808 01:51:27.702159 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.383: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.704831 7378 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:27.706104 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.708416 7339 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.710899 5212 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.726804 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.736727 7378 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.747646 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.751452 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.758172 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 387: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.761209 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 382: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.761088 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.763458 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 383: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.762814 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.771059 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.771865 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.777892 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.779686 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 343 to 344, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.781265 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 382 to 383, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.784631 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.786070 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.787397 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.800788 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:27.804674 5212 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 1.352, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.819913 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 1.388, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.813982 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 2.354, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.826704 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 2.383, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.830199 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20260808 01:51:27.833863 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 383. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20260808 01:51:27.815698 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 2.350, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.834124 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20260808 01:51:27.836382 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 354. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20260808 01:51:27.839514 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:27.841056 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20260808 01:51:27.844564 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20260808 01:51:27.819150 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 3.344, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:27.845374 7339 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.820677 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.853554 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 352. Preceding OpId from leader: term: 1 index: 353. (index mismatch)
I20260808 01:51:27.855657 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:27.855875 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20260808 01:51:27.856263 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.857087 7378 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:27.857658 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.858441 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
W20260808 01:51:27.860391 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.860805 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.863761 5068 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.864913 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.866928 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 344. Preceding OpId from leader: term: 3 index: 345. (index mismatch)
I20260808 01:51:27.870513 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.879926 7367 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20260808 01:51:27.865836 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20260808 01:51:27.872730 7378 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.892858 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.894843 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.885061 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.384: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.901726 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.907080 7379 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.909905 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 1.384, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.911412 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.912173 5068 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.915446 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.919584 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 383: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.922708 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:27.924671 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.925084 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.925812 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 388: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:51:27.926563 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 354 but the committed config has opid_index of 355
W20260808 01:51:27.927222 4770 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER failed with leader 99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 350 but the committed config has opid_index of 351
I20260808 01:51:27.929353 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:27.929731 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:27.928632 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.932845 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20260808 01:51:27.934927 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.001s
I20260808 01:51:27.935442 7378 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.939478 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.941481 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20260808 01:51:27.945333 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.345: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.946717 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.957842 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 344: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:27.963063 7378 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:27.964466 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.974174 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 384: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.975400 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.953346 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.986967 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.988307 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:27.992374 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:27.993489 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:27.989334 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 383 to 384, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.006362 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.008690 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.020536 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 389, Committed index: 389, Last appended: 1.389, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.025034 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.026618 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 2.384, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.029811 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20260808 01:51:28.032725 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.033125 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.033143 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.001s
I20260808 01:51:28.033831 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.034163 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.040548 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.040992 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.042649 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.050112 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.053426 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20260808 01:51:28.046227 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 384. Preceding OpId from leader: term: 2 index: 385. (index mismatch)
I20260808 01:51:28.056473 7379 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.001s
I20260808 01:51:28.057273 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.057653 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.065896 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20260808 01:51:28.070761 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.001s
I20260808 01:51:28.075284 7379 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.001s
I20260808 01:51:28.079602 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.087606 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.094919 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 2.355, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.097790 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.348, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.097750 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.103608 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 389: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.107390 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.107757 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.111864 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.120411 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 355. Preceding OpId from leader: term: 2 index: 356. (index mismatch)
I20260808 01:51:28.121655 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 348. Preceding OpId from leader: term: 2 index: 349. (index mismatch)
I20260808 01:51:28.120560 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.133417 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:28.134802 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.135654 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 384: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.135900 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:28.136474 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.137428 7367 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.139781 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.141978 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 344 to 345, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.142561 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.144403 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.385: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.160465 7379 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.170569 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.175129 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 1.354, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.179497 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.179941 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.180873 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.184689 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 3.345, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.186251 5068 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20260808 01:51:28.188449 5068 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20260808 01:51:28.188308 7339 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:28.191215 7339 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:28.195525 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.197983 7378 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.200680 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.206051 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20260808 01:51:28.207883 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20260808 01:51:28.209084 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.209534 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
W20260808 01:51:28.210721 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.212733 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.214828 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.216022 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 384 to 385, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.201392 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.220485 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.223239 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 345. Preceding OpId from leader: term: 3 index: 346. (index mismatch)
I20260808 01:51:28.220654 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.229462 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.237588 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.240190 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.235836 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 2.385, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.243387 7378 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.002s
I20260808 01:51:28.241487 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.248324 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.254136 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 385. Preceding OpId from leader: term: 2 index: 386. (index mismatch)
I20260808 01:51:28.255529 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20260808 01:51:28.259334 4782 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.259388 7367 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.261940 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.264578 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.265842 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.267983 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.258864 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 1.390, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.274533 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.346: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.275638 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20260808 01:51:28.277518 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 385: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.278280 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20260808 01:51:28.281806 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.285656 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.386: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.293175 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 345: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.293813 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.295292 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 385: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.297503 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.295194 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 2.356, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.301075 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.302767 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 1.355, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.307587 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 356. Preceding OpId from leader: term: 2 index: 357. (index mismatch)
I20260808 01:51:28.309350 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.309222 7367 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
W20260808 01:51:28.309753 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.311121 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.311991 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.316490 7368 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.318342 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20260808 01:51:28.319308 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.319842 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:28.326893 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20260808 01:51:28.328248 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20260808 01:51:28.328130 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 345 to 346, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.331897 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 390: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.336377 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.339443 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.344899 7367 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.347609 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.351184 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 386: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.351778 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.362063 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.365756 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.365729 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.366191 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 385 to 386, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.370205 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 3.346, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.385922 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20260808 01:51:28.386129 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 346. Preceding OpId from leader: term: 3 index: 347. (index mismatch)
I20260808 01:51:28.388731 7368 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:28.389529 7378 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20260808 01:51:28.392834 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 2.349, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.399313 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.400172 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
W20260808 01:51:28.400998 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.401823 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.402755 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.404273 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 2.386, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.405028 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:28.406356 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.406942 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.408680 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.409973 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.411074 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.412114 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 349. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260808 01:51:28.411907 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.347: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.417048 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:28.419903 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.420284 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.419610 7367 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.429386 7339 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.432632 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 1.356, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.434273 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.440382 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 346: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.442160 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.431596 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 386. Preceding OpId from leader: term: 2 index: 387. (index mismatch)
I20260808 01:51:28.443852 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20260808 01:51:28.446560 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20260808 01:51:28.448141 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.446517 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 2.357, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.451591 7338 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:28.453838 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.455276 5075 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.460285 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.460748 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.462976 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 1.391, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.452678 7323 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.466550 5072 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
I20260808 01:51:28.468892 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.470307 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20260808 01:51:28.469846 7379 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.471872 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20260808 01:51:28.469219 7338 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:28.477972 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.482440 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.487136 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.488538 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.490703 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.491087 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.491830 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20260808 01:51:28.493247 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20260808 01:51:28.501781 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 387: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.503062 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.503641 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.504900 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.511514 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 391: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.511917 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.387: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.515434 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.516959 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.518452 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 346 to 347, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.523249 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.523156 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.525127 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 386: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.526198 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.527271 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.531464 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.539862 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.544627 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.549602 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 386 to 387, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.552151 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.583974 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 1.388, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.585160 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 2.387, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.590896 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.591320 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.594996 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.602144 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.605111 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 387. Preceding OpId from leader: term: 2 index: 388. (index mismatch)
I20260808 01:51:28.607791 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20260808 01:51:28.610266 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 1 index: 393. (index mismatch)
I20260808 01:51:28.611996 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 3.347, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.612916 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.613637 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:28.612010 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 2.354, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.623028 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 354. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20260808 01:51:28.623847 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20260808 01:51:28.625154 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20260808 01:51:28.626394 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20260808 01:51:28.631954 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20260808 01:51:28.633417 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:28.639516 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:28.645812 7234 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.652890 7379 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.653091 7378 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.661087 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.666278 5075 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.667021 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.675577 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.678543 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.388: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.691052 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:28.695997 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 347. Preceding OpId from leader: term: 3 index: 348. (index mismatch)
I20260808 01:51:28.697099 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.699941 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 388: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.711978 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.714933 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.719712 5212 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 2.350, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.720587 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.720810 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.721333 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 387: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.723264 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.724095 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 392: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.725756 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.741582 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
I20260808 01:51:28.742378 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:28.743291 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:28.745908 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 2 index: 351. (index mismatch)
I20260808 01:51:28.747233 7378 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:28.749209 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20260808 01:51:28.751370 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.753090 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:28.755863 7339 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.763640 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.767176 7379 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.768770 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.773022 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.796208 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.798177 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 387 to 388, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.800673 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:28.806936 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:51:28.807349 5149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.180.68:35651, user_credentials={real_user=slave}} blocked reactor thread for 86664.8us
I20260808 01:51:28.808555 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.808939 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.809265 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.809248 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.809983 7368 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.001s
I20260808 01:51:28.813730 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.820973 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.823105 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 2.388, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.823407 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.348: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.826206 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.826785 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.827848 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.829962 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.830338 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.833472 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 347: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.838874 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 388. Preceding OpId from leader: term: 2 index: 389. (index mismatch)
I20260808 01:51:28.840142 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 389, Committed index: 389, Last appended: 1.389, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.840526 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20260808 01:51:28.844023 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.844851 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20260808 01:51:28.846940 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.847771 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
I20260808 01:51:28.848780 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.852578 5192 tablet_service.cc:1467] Tablet server has 6 leaders and 0 scanners
I20260808 01:51:28.855656 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.857744 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 388: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.848379 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.853595 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.389: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.861981 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 347 to 348, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.862438 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.865490 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 1.393, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.875015 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 389: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.876237 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:28.879571 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 393. Preceding OpId from leader: term: 1 index: 394. (index mismatch)
I20260808 01:51:28.880692 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.881058 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.883105 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s
I20260808 01:51:28.891080 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.900044 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.906183 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 388 to 389, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:28.910368 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 393: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.922941 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 2.359, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.930037 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.930291 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 359. Preceding OpId from leader: term: 2 index: 360. (index mismatch)
W20260808 01:51:28.930415 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.932539 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 389, Committed index: 389, Last appended: 2.389, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.934965 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 2.355, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.938526 7379 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:28.940595 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 1.358, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.945176 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20260808 01:51:28.945155 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.947281 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:28.946707 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
W20260808 01:51:28.947633 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.948133 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 355. Preceding OpId from leader: term: 2 index: 356. (index mismatch)
I20260808 01:51:28.948585 7338 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:28.949530 7367 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:28.953703 7367 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.954721 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 2.351, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.956339 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 2.354, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.961148 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:28.961524 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:28.968586 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 351. Preceding OpId from leader: term: 2 index: 352. (index mismatch)
I20260808 01:51:28.968708 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 3.348, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:28.969897 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 354. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20260808 01:51:28.970727 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 389. Preceding OpId from leader: term: 2 index: 390. (index mismatch)
I20260808 01:51:28.972046 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
I20260808 01:51:28.972054 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:28.974035 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.976703 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:28.968818 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:28.978324 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:28.980178 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.981721 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.390: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:28.991803 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 348. Preceding OpId from leader: term: 3 index: 349. (index mismatch)
I20260808 01:51:28.993677 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 1.390, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:28.998980 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.002319 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 389: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.003047 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20260808 01:51:29.003351 7403 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20260808 01:51:29.004428 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.004590 7379 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.009449 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20260808 01:51:29.010964 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.010993 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.013588 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.014106 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.014900 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:28.999722 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 1.394, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.017247 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.021744 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 394. Preceding OpId from leader: term: 1 index: 395. (index mismatch)
I20260808 01:51:29.024565 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I20260808 01:51:29.025460 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.030606 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.033471 7368 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.034404 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.349: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.036664 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.039067 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.043073 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.044623 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.046259 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.055409 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.066780 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 394: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.068848 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 390: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.075318 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 348: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.077802 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.079342 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.080451 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 389 to 390, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.085230 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.096558 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.116529 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 1.391, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.120692 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20260808 01:51:29.122062 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
W20260808 01:51:29.124789 4818 debug-util.cc:398] Leaking SignalData structure 0x7b0800050080 after lost signal to thread 4749
I20260808 01:51:29.125301 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 2.360, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.130595 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.132289 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 2.356, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.133198 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.142812 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 356. Preceding OpId from leader: term: 2 index: 357. (index mismatch)
I20260808 01:51:29.144595 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 2.390, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.156687 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:29.157347 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 390. Preceding OpId from leader: term: 2 index: 391. (index mismatch)
I20260808 01:51:29.158841 6972 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20260808 01:51:29.160437 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:29.145503 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 360. Preceding OpId from leader: term: 2 index: 361. (index mismatch)
W20260808 01:51:29.161290 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.162550 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:29.166016 7339 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.167543 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.167771 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.171234 5210 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.391: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.176172 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.178187 7367 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.183051 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.185708 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 390: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.196136 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 391: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.196902 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.204459 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.211645 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 348 to 349, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.212996 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.214046 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.215235 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.232945 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.248585 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.262012 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.262421 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.264449 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 1 index: 393. (index mismatch)
I20260808 01:51:29.265910 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20260808 01:51:29.275736 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.279043 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.279073 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 2.361, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.283191 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 1.359, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.290467 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 361. Preceding OpId from leader: term: 2 index: 362. (index mismatch)
I20260808 01:51:29.291676 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 2.355, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.292636 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:29.295328 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 359. Preceding OpId from leader: term: 1 index: 360. (index mismatch)
I20260808 01:51:29.297995 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.299649 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.299737 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:29.302783 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 355. Preceding OpId from leader: term: 2 index: 356. (index mismatch)
I20260808 01:51:29.303609 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 1.395, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.309900 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:29.316959 7339 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.317896 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.318264 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.318643 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.320176 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.320387 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.325953 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.328547 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:29.323050 7378 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.329502 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 3.349, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.331063 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20260808 01:51:29.329532 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 2.352, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.334317 7368 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.352115 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 392: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:51:29.357698 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.359630 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.361384 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:29.332677 6972 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20260808 01:51:29.336383 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 349. Preceding OpId from leader: term: 3 index: 350. (index mismatch)
I20260808 01:51:29.336103 5075 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.337896 5074 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 352. Preceding OpId from leader: term: 2 index: 353. (index mismatch)
I20260808 01:51:29.371408 6972 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.374076 7367 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
W20260808 01:51:29.376019 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.381718 7338 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260808 01:51:29.375201 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.393194 7367 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.403965 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.350: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.404439 7404 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.411731 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.428241 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.430217 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 349: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.431701 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.432335 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 395: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.441730 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.443332 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.444491 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 390 to 391, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.445560 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.450512 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.469969 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.471525 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.472879 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.476855 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.477891 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 349 to 350, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.478221 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 2.362, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.483233 5075 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 362. Preceding OpId from leader: term: 2 index: 363. (index mismatch)
I20260808 01:51:29.484949 7404 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:29.485426 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 2.357, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.493243 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:29.493242 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
W20260808 01:51:29.493607 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.495112 7378 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:29.501812 7404 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.507424 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.505827 7378 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.509756 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.527527 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 1.360, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.532001 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 2.356, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.534021 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 3.350, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.540959 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 2.353, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.545174 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 1.393, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.549197 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 356. Preceding OpId from leader: term: 2 index: 357. (index mismatch)
I20260808 01:51:29.550673 5072 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 360. Preceding OpId from leader: term: 1 index: 361. (index mismatch)
I20260808 01:51:29.551836 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 350. Preceding OpId from leader: term: 3 index: 351. (index mismatch)
I20260808 01:51:29.556084 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 2.391, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.556504 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.560462 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.560400 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 1.396, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.571754 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.572229 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.575623 7404 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:29.573267 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.576898 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:29.589346 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 353. Preceding OpId from leader: term: 2 index: 354. (index mismatch)
I20260808 01:51:29.577979 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20260808 01:51:29.578665 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 391. Preceding OpId from leader: term: 2 index: 392. (index mismatch)
I20260808 01:51:29.591588 7404 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:29.591557 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.591954 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20260808 01:51:29.593189 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20260808 01:51:29.578915 7367 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20260808 01:51:29.594496 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.579730 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 393. Preceding OpId from leader: term: 1 index: 394. (index mismatch)
I20260808 01:51:29.600176 6972 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s
I20260808 01:51:29.605093 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.610538 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.612232 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.620543 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.620949 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.631188 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.637984 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.639927 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.634898 7404 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.635668 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.642236 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.392: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.650269 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 396: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.648820 7368 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.656443 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 391: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.657784 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
I20260808 01:51:29.662492 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.638096 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.662998 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.667244 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 393: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.672366 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.676342 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.677886 7367 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:29.690899 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 2.357, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.660796 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.639358 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 2.363, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.704026 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.704392 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.708433 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.708792 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.713397 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.351: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.714282 5068 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 363. Preceding OpId from leader: term: 2 index: 364. (index mismatch)
I20260808 01:51:29.719565 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.720868 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.723872 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 350 to 351, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.726159 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.729480 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 1.397, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.732854 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7704560c94d24a499a485f316d701990 | 127.29.180.126:32885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
I20260808 01:51:29.740455 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked
I20260808 01:51:29.742573 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
01adc8a4bde24946b6bf7e18e876e977 | 127.29.180.65:33111 | HEALTHY | <none> | 0 | 0
99d0ee4efbd948a3b143ff2297ad9fc9 | 127.29.180.67:46093 | HEALTHY | <none> | 6 | 0
c37035739bf347d49e5ed4b2f884adac | 127.29.180.68:35651 | HEALTHY | <none> | 0 | 0
ec65d57b7135430eb6695032355bd356 | 127.29.180.66:43479 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
I20260808 01:51:29.745443 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.745849 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
cfile_inject_corruption | 0 | hidden | all 4 server(s) checked
enable_tablet_copy | true | unsafe | all 4 server(s) checked
env_inject_eio | 0 | hidden | all 4 server(s) checked
flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked
flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.29.180.65 | experimental | 127.29.180.65:33111
I20260808 01:51:29.723546 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.024s
local_ip_for_outbound_sockets | 127.29.180.66 | experimental | 127.29.180.66:43479
local_ip_for_outbound_sockets | 127.29.180.67 | experimental | 127.29.180.67:46093
local_ip_for_outbound_sockets | 127.29.180.68 | experimental | 127.29.180.68:35651
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-0/data-0/info.pb | hidden | 127.29.180.65:33111
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-1/data-0/info.pb | hidden | 127.29.180.66:43479
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-2/data-0/info.pb | hidden | 127.29.180.67:46093
server_dump_info_path | /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/data-0/info.pb | hidden | 127.29.180.68:35651
tablet_inject_kudu_2233 | 0 | hidden,unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
I20260808 01:51:29.746805 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.180.84:33941 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
I20260808 01:51:29.750492 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 391 to 392, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d7aa70246b445149411741e595f27c2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 2 | 358 | Yes
B | A B* | 2 | 359 | Yes
C | [config not available] | | |
I20260808 01:51:29.754508 7339 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Tablet 9ee6f0c29217471e93394d4b9814144e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 1 | 393 | Yes
B | A* B | 1 | 393 | Yes
C | [config not available] | | |
Tablet 1fff9439a0bd4286827363f69521c44f of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 3 | 347 | Yes
B | A B* C~ | 3 | 348 | Yes
C | [config not available] | | |
Tablet 86d58fdfa95b45f68c0648a585924d05 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
I20260808 01:51:29.758903 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B | 2 | 388 | Yes
B | A* B C~ | 2 | 389 | Yes
C | [config not available] | | |
Tablet 5843450fd1f54792899cc7190c94821f of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 1 | 358 | Yes
B | A B* C~ | 1 | 358 | Yes
Tablet 8d367581a54c433ea4f227bd4f479ed2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28)
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
I20260808 01:51:29.767504 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.772892 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.774333 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.775269 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.777365 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20260808 01:51:29.778811 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20260808 01:51:29.785588 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* | 2 | 353 | Yes
B | A B* C~ | 2 | 354 | Yes
C | [config not available] | | |
Tablet 2a97be89f618431b90b49c97b2c9f001 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING [LEADER]
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
I20260808 01:51:29.794116 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C~ | 1 | 389 | Yes
B | A* B | 1 | 390 | Yes
Tablet 0c44eefb43024e9290fdba8647e745fe of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
ec65d57b7135430eb6695032355bd356 (127.29.180.66:43479): RUNNING
99d0ee4efbd948a3b143ff2297ad9fc9 (127.29.180.67:46093): RUNNING [LEADER]
All reported replicas are:
A = ec65d57b7135430eb6695032355bd356
B = 99d0ee4efbd948a3b143ff2297ad9fc9
C = c37035739bf347d49e5ed4b2f884adac
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 2 | 350 | Yes
B | A B* | 2 | 351 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 10 | 2 | 0 | 0 | 8
I20260808 01:51:29.796746 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 6
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 10
Replicas | 26
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260808 01:51:29.800208 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.805362 30417 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260808 01:51:29.808154 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 397: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.814167 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 2.354, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.819001 5069 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 354. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20260808 01:51:29.820142 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 2.392, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.820549 7339 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:29.820675 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.822211 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.824823 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.827085 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.827734 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 392. Preceding OpId from leader: term: 2 index: 393. (index mismatch)
I20260808 01:51:29.827244 7338 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.829456 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.829870 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.831069 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20260808 01:51:29.835098 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 3.351, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.840401 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 1.394, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.840440 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.841467 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.841846 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.837409 5069 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.844946 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 351. Preceding OpId from leader: term: 3 index: 352. (index mismatch)
I20260808 01:51:29.848603 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.849000 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.849798 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 392: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.850299 7404 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I20260808 01:51:29.856681 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.857882 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.857583 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 394. Preceding OpId from leader: term: 1 index: 395. (index mismatch)
I20260808 01:51:29.858186 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.862334 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 2.364, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.857365 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.393: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.867697 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 1.361, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.871532 5206 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 2.359, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.876078 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.877081 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.877067 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I20260808 01:51:29.876509 5074 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.352: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.879926 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.880726 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.884163 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.885717 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.893225 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 364. Preceding OpId from leader: term: 2 index: 365. (index mismatch)
I20260808 01:51:29.893823 5069 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 361. Preceding OpId from leader: term: 1 index: 362. (index mismatch)
I20260808 01:51:29.894977 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 351: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.895287 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 359. Preceding OpId from leader: term: 2 index: 360. (index mismatch)
I20260808 01:51:29.896950 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:29.897269 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 394: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.898818 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:29.901044 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:29.896162 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 392 to 393, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 393 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.905328 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.915109 7368 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.917367 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.919082 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 351 to 352, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 352 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:29.920917 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
I20260808 01:51:29.920750 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.921244 7338 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.928977 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 2.393, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.933471 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 2.355, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:29.937402 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 393. Preceding OpId from leader: term: 2 index: 394. (index mismatch)
I20260808 01:51:29.938738 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:29.939131 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:29.939108 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s
I20260808 01:51:29.937809 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 352, Committed index: 352, Last appended: 3.352, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.938895 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.940848 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.944790 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 355. Preceding OpId from leader: term: 2 index: 356. (index mismatch)
I20260808 01:51:29.945401 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.946292 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.947085 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.950687 7368 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:29.950755 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.394: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.954044 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:29.962641 7368 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.966209 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.971153 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 1.398, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:29.976768 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 393: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.978029 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:29.977962 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.979697 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:29.989439 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:29.994572 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 398. Preceding OpId from leader: term: 1 index: 399. (index mismatch)
I20260808 01:51:29.995983 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I20260808 01:51:29.997819 5072 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 352. Preceding OpId from leader: term: 3 index: 353. (index mismatch)
I20260808 01:51:29.998857 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:29.988740 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.000537 7338 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 353, Last known committed idx: 352, Time since last communication: 0.000s
I20260808 01:51:30.002420 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.006699 7378 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.008538 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.353: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.012738 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.015977 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 352: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.024976 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 398: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.028904 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.031551 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.034584 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 1.395, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.039054 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20260808 01:51:30.040548 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s
I20260808 01:51:30.048141 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.053134 5071 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch)
I20260808 01:51:30.052923 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.054793 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:30.059029 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.059993 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 2.360, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.063467 7378 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.067171 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.074666 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.075876 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 352 to 353, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 353 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.077190 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 393 to 394, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.078474 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.079464 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.080649 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.083952 5072 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 360. Preceding OpId from leader: term: 2 index: 361. (index mismatch)
I20260808 01:51:30.085422 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 395: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.085354 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:30.088291 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.091305 7338 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.095429 5072 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.106208 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 2.394, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.107869 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.111524 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.112715 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.113066 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.115312 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.117594 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.121088 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 394. Preceding OpId from leader: term: 2 index: 395. (index mismatch)
I20260808 01:51:30.122555 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I20260808 01:51:30.129133 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.130538 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.395: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.143093 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.143321 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 394: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:51:30.143489 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.146095 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20260808 01:51:30.147560 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20260808 01:51:30.148877 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.153245 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.163597 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.164572 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 399: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.165570 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 3.353, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.170868 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 353. Preceding OpId from leader: term: 3 index: 354. (index mismatch)
I20260808 01:51:30.171152 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.171478 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.172282 7378 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20260808 01:51:30.175974 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 1.396, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.178679 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.179776 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.354: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.184823 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 2.356, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.189409 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.189797 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.188835 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 396. Preceding OpId from leader: term: 1 index: 397. (index mismatch)
I20260808 01:51:30.191951 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 1.363, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.198616 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20260808 01:51:30.199862 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 356. Preceding OpId from leader: term: 2 index: 357. (index mismatch)
I20260808 01:51:30.197739 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 2.365, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.201113 7378 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:30.205384 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 365. Preceding OpId from leader: term: 2 index: 366. (index mismatch)
I20260808 01:51:30.206074 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.207015 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
W20260808 01:51:30.206457 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.211892 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.212229 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 2.359, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.217419 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.217772 7368 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.220707 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.221092 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.223086 5072 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.226037 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 359. Preceding OpId from leader: term: 2 index: 360. (index mismatch)
I20260808 01:51:30.227618 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:30.232241 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.234846 4782 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 394 to 395, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.237030 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.238794 7378 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.240429 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.242280 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
W20260808 01:51:30.246402 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.249841 5072 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 2.395, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.253890 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 395. Preceding OpId from leader: term: 2 index: 396. (index mismatch)
I20260808 01:51:30.251297 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.255354 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.001s
I20260808 01:51:30.257469 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.257939 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 1.400, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.261030 6972 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.270504 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.272750 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 363. Preceding OpId from leader: term: 1 index: 364. (index mismatch)
I20260808 01:51:30.272418 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 353 to 354, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.273841 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 353: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.274555 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.275568 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 396: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.276175 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.277710 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:30.279309 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.284720 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.396: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.288868 7368 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.290913 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 395: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.290802 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.295264 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20260808 01:51:30.297076 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20260808 01:51:30.303769 7323 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.305382 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.309875 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.319342 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 400: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.325719 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 3.354, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.330546 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 354. Preceding OpId from leader: term: 3 index: 355. (index mismatch)
I20260808 01:51:30.331266 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 2.366, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.332088 7368 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260808 01:51:30.333714 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 2.361, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.335569 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 366. Preceding OpId from leader: term: 2 index: 367. (index mismatch)
I20260808 01:51:30.337896 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.340190 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 361. Preceding OpId from leader: term: 2 index: 362. (index mismatch)
I20260808 01:51:30.340353 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.340731 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.341742 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:30.341982 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.343652 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:30.343331 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.344779 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 395 to 396, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.346189 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.353739 7378 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.353874 7367 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.364392 5072 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 1.397, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.365253 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.355: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.368188 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 354: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.367926 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.368960 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.370671 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 2.396, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.374276 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20260808 01:51:30.380049 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.381986 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.383241 5211 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 396. Preceding OpId from leader: term: 2 index: 397. (index mismatch)
I20260808 01:51:30.386130 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 1.364, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.386449 5071 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.383248 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 2.360, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.388259 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 2.357, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
W20260808 01:51:30.389686 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.390456 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.390805 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.397274 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20260808 01:51:30.398558 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I20260808 01:51:30.401087 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.401453 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.401726 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
I20260808 01:51:30.403592 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:30.404490 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.412845 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20260808 01:51:30.413215 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 360. Preceding OpId from leader: term: 2 index: 361. (index mismatch)
I20260808 01:51:30.415544 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.418534 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.419351 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:30.419468 7406 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.001s
I20260808 01:51:30.422683 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.423813 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 364. Preceding OpId from leader: term: 1 index: 365. (index mismatch)
I20260808 01:51:30.427315 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.431878 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:30.438012 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.433157 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.445377 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 354 to 355, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.446970 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.448383 7367 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.453459 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.454134 5072 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.470114 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.474720 5211 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 3.355, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.478832 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 2.362, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.480912 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.483000 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.486224 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.487488 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.491294 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 362. Preceding OpId from leader: term: 2 index: 363. (index mismatch)
I20260808 01:51:30.491464 7406 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.493865 7367 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.001s
I20260808 01:51:30.504002 7367 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.507580 4782 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.508873 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.509600 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.517060 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.525882 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.530805 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.531224 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.535452 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 397: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.543133 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.544600 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.548285 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 355. Preceding OpId from leader: term: 3 index: 356. (index mismatch)
I20260808 01:51:30.548303 7326 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.549839 7367 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20260808 01:51:30.560422 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.397: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.571641 7368 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.573076 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 1.398, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.579082 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.579473 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.580904 5072 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.356: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.583828 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 398. Preceding OpId from leader: term: 1 index: 399. (index mismatch)
I20260808 01:51:30.583829 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 396: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.586118 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I20260808 01:51:30.600234 7406 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.601460 5206 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.608883 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 401: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.614123 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 398: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.598753 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 2.367, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.617595 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.619942 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 355: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.621377 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.622927 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 396 to 397, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 397 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.624658 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.624603 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 367. Preceding OpId from leader: term: 2 index: 368. (index mismatch)
W20260808 01:51:30.625329 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.629014 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 2.361, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.629676 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:30.634862 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.634807 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
W20260808 01:51:30.635202 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.636552 7378 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:30.640357 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.643878 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 2.397, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.637902 5072 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 361. Preceding OpId from leader: term: 2 index: 362. (index mismatch)
I20260808 01:51:30.647282 7378 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.651067 5206 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 1 index: 400. (index mismatch)
I20260808 01:51:30.651808 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 397. Preceding OpId from leader: term: 2 index: 398. (index mismatch)
I20260808 01:51:30.652163 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:30.654760 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.655501 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20260808 01:51:30.657927 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20260808 01:51:30.661278 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.645097 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.662704 4782 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 355 to 356, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.663883 7406 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.667043 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.672106 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.398: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.677229 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 397: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.679840 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.679678 7282 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.680800 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.686292 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.689561 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.690342 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 399: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.690922 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.691778 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.698047 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 1.402, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.703491 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 402. Preceding OpId from leader: term: 1 index: 403. (index mismatch)
I20260808 01:51:30.704816 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.705030 7406 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I20260808 01:51:30.705823 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 397 to 398, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.706965 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.708236 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.711128 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 2.363, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.716403 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 363. Preceding OpId from leader: term: 2 index: 364. (index mismatch)
I20260808 01:51:30.718611 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:30.719085 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 3.356, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.723104 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 1.365, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.723822 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:30.723830 5071 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 356. Preceding OpId from leader: term: 3 index: 357. (index mismatch)
W20260808 01:51:30.724153 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.726138 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20260808 01:51:30.727562 7282 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.728415 7406 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.729738 5071 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.732555 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.732934 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.732575 5211 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.737568 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 365. Preceding OpId from leader: term: 1 index: 366. (index mismatch)
I20260808 01:51:30.738039 7338 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.740095 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 402: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.740995 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:30.742029 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.357: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.745544 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.747325 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 356: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.751425 7378 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.752823 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.754138 4782 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.760366 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.764274 5071 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 2.398, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.768453 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 1.400, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.769990 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 2.368, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.775537 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.777974 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.778993 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 398. Preceding OpId from leader: term: 2 index: 399. (index mismatch)
I20260808 01:51:30.779086 5211 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 2.359, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.781002 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.781363 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.783604 5071 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 368. Preceding OpId from leader: term: 2 index: 369. (index mismatch)
I20260808 01:51:30.784116 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I20260808 01:51:30.784495 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.785857 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.778997 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 2.362, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.786962 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.788165 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 356 to 357, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.789961 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 400. Preceding OpId from leader: term: 1 index: 401. (index mismatch)
I20260808 01:51:30.791460 7406 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20260808 01:51:30.793968 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.794308 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.792999 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.794971 7338 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20260808 01:51:30.796183 5206 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.399: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.803479 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.806485 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 398: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.806677 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.807693 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 362. Preceding OpId from leader: term: 2 index: 363. (index mismatch)
I20260808 01:51:30.808938 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:30.809437 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.815632 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 400: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.814910 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.815151 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.817806 5072 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 359. Preceding OpId from leader: term: 2 index: 360. (index mismatch)
I20260808 01:51:30.819435 5071 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.820015 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:30.829191 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.832861 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.837796 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.839633 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 368: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.841269 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.855031 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 2.364, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.859884 5074 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 364. Preceding OpId from leader: term: 2 index: 365. (index mismatch)
I20260808 01:51:30.861081 7378 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:30.864190 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.865384 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.868561 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 398 to 399, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.870393 5206 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 3.357, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.870497 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.874692 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:30.876104 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.877334 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20260808 01:51:30.877956 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.878831 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:30.879271 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.880335 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.880692 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.877458 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.888581 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 357. Preceding OpId from leader: term: 3 index: 358. (index mismatch)
I20260808 01:51:30.893379 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.895129 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.895478 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.896454 5206 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 1 index: 404. (index mismatch)
I20260808 01:51:30.897895 7406 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20260808 01:51:30.902421 7338 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20260808 01:51:30.904047 7409 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.907755 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.911131 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 2.363, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.912238 5211 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 2.369, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.911975 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.915890 5206 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 2.360, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.920181 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.920567 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.921489 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.921823 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.926201 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20260808 01:51:30.927402 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.931233 5069 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 2.399, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.931183 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:30.933192 5071 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:30.936190 5208 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 399. Preceding OpId from leader: term: 2 index: 400. (index mismatch)
I20260808 01:51:30.938056 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20260808 01:51:30.943647 7406 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.944749 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch)
I20260808 01:51:30.945354 5206 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 1.367, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:30.950089 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 403: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:30.950918 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 363. Preceding OpId from leader: term: 2 index: 364. (index mismatch)
I20260808 01:51:30.955448 7410 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.001s
I20260808 01:51:30.960357 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.400: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.963126 7366 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.967509 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:30.967806 7378 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:30.967726 7367 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.969244 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.972983 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.974119 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:30.974661 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.980324 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 367. Preceding OpId from leader: term: 1 index: 368. (index mismatch)
I20260808 01:51:30.981876 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:30.982434 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 401: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:30.985411 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 360. Preceding OpId from leader: term: 2 index: 361. (index mismatch)
I20260808 01:51:30.973006 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:30.989253 7338 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.993916 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.998021 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.002655 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:30.999922 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.005791 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:31.012575 7404 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.013392 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 399: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.021315 7339 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.024317 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.030861 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.404, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.034248 5072 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.035467 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 1 index: 405. (index mismatch)
I20260808 01:51:31.037164 7406 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.001s
I20260808 01:51:31.039248 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 2.365, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.043825 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.045680 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.046036 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.047060 5206 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.053561 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.058801 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 365. Preceding OpId from leader: term: 2 index: 366. (index mismatch)
I20260808 01:51:31.063860 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 404: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.066691 7378 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:31.073146 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.075606 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 369: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.080189 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.083475 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 399 to 400, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 400 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.086381 5071 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.358: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.087307 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.087921 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.088974 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.097548 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.098822 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 357: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.101456 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 2.370, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.104970 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.107249 5208 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.112645 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.114269 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 370. Preceding OpId from leader: term: 2 index: 371. (index mismatch)
I20260808 01:51:31.117817 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.118899 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20260808 01:51:31.126957 7404 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.133385 5072 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 1.405, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.134892 4786 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.138685 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.141192 5068 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 1.402, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.143968 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
I20260808 01:51:31.146386 7366 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20260808 01:51:31.142591 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20260808 01:51:31.147732 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 2.400, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.150086 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20260808 01:51:31.151284 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.151615 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.148272 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 357 to 358, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.154217 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 402. Preceding OpId from leader: term: 1 index: 403. (index mismatch)
I20260808 01:51:31.160573 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I20260808 01:51:31.162536 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.162871 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.162411 7409 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.163080 7282 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.163842 5206 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 400. Preceding OpId from leader: term: 2 index: 401. (index mismatch)
I20260808 01:51:31.167626 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.167980 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.170153 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.171674 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20260808 01:51:31.179621 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 370: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.186611 7406 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.172211 5071 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.201649 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 2.361, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.195254 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.401: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.206777 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 3.358, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.207504 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.211661 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 368: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.214247 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.211042 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.216538 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.217648 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 405: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.220057 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.220445 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.226590 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 400: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.227928 5071 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 361. Preceding OpId from leader: term: 2 index: 362. (index mismatch)
I20260808 01:51:31.229462 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 402: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.229387 7404 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.001s
I20260808 01:51:31.230791 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 358. Preceding OpId from leader: term: 3 index: 359. (index mismatch)
I20260808 01:51:31.231907 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.232682 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 2.366, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.237726 7338 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.237872 5071 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 366. Preceding OpId from leader: term: 2 index: 367. (index mismatch)
I20260808 01:51:31.239135 7367 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20260808 01:51:31.240332 7367 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:31.238741 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 2.364, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.245074 5074 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.250167 5071 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 364. Preceding OpId from leader: term: 2 index: 365. (index mismatch)
I20260808 01:51:31.252264 7404 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:31.253721 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.258692 5069 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.255820 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 2.371, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.255656 7339 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.263891 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.278923 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.280511 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.282711 7404 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.282979 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.284248 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.359: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.284468 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 400 to 401, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.287760 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.288723 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.294450 5069 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 371. Preceding OpId from leader: term: 2 index: 372. (index mismatch)
I20260808 01:51:31.296114 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260808 01:51:31.295610 7368 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.302141 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.304384 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.308401 5071 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 1.406, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.311900 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.312183 5208 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 2.362, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.319178 5068 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 362. Preceding OpId from leader: term: 2 index: 363. (index mismatch)
I20260808 01:51:31.319195 5075 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch)
I20260808 01:51:31.320135 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.321050 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.321388 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.321678 7367 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:31.323787 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20260808 01:51:31.328233 5069 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.330541 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20260808 01:51:31.330313 4786 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.331758 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.323750 7367 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:31.324373 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.333254 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.337131 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.407: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.338533 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 358 to 359, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.345112 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.349924 7367 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.354203 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.355135 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.360445 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.363639 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.367240 5211 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 1 index: 404. (index mismatch)
I20260808 01:51:31.368765 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20260808 01:51:31.368615 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 401, Committed index: 401, Last appended: 2.401, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.373448 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 401. Preceding OpId from leader: term: 2 index: 402. (index mismatch)
I20260808 01:51:31.374768 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 371: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.374961 7406 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.376003 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 401, Time since last communication: 0.000s
I20260808 01:51:31.376777 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.385996 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 369: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.385952 5210 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 2.365, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.393312 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.393689 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.393160 5211 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 2.367, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.397408 4784 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.398736 4784 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.401167 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 367. Preceding OpId from leader: term: 2 index: 368. (index mismatch)
I20260808 01:51:31.395895 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 359, Committed index: 359, Last appended: 3.359, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.403024 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 403: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.400913 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.403982 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.404518 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:31.373071 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.411320 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.411778 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.411779 5068 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 359. Preceding OpId from leader: term: 3 index: 360. (index mismatch)
I20260808 01:51:31.412302 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.413205 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 360, Last known committed idx: 359, Time since last communication: 0.000s
I20260808 01:51:31.419837 7368 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.424743 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 406: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.425712 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 401: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.427783 7404 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.431749 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.402: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.432719 4784 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.434736 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.436105 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.437928 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 365. Preceding OpId from leader: term: 2 index: 366. (index mismatch)
I20260808 01:51:31.437755 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.360: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.439949 7282 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:31.436218 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.448239 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.457250 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.460675 5069 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.463889 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 2.363, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.464571 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 1.370, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.469731 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.470134 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.471760 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 363. Preceding OpId from leader: term: 2 index: 364. (index mismatch)
I20260808 01:51:31.473059 7338 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:31.480490 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 371. (index mismatch)
I20260808 01:51:31.481916 7338 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20260808 01:51:31.486033 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.404, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.486529 5074 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 407, Committed index: 407, Last appended: 1.407, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.493912 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.494257 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.494617 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20260808 01:51:31.495988 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.000s
I20260808 01:51:31.492357 7338 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.504644 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.505681 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.511458 7366 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.408: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.513770 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.530526 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 363: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.529772 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.534843 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 407: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.538901 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 2.372, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.545328 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 359: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.545939 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 372. Preceding OpId from leader: term: 2 index: 373. (index mismatch)
I20260808 01:51:31.549252 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.549633 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.548468 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.002s
I20260808 01:51:31.551781 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 370: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.560349 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.569065 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.571259 5210 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 1 index: 405. (index mismatch)
I20260808 01:51:31.572472 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.578172 7366 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20260808 01:51:31.581983 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.584208 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.586107 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.587360 7406 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.589190 5207 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.598001 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 404: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.597672 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 359 to 360, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 360 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.610890 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 372: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.621456 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 401 to 402, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.623394 4786 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.630716 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 2.368, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.634930 5073 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 368. Preceding OpId from leader: term: 2 index: 369. (index mismatch)
I20260808 01:51:31.636432 7282 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20260808 01:51:31.637405 5207 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 2.364, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.638113 5073 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 2.402, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.642360 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 364. Preceding OpId from leader: term: 2 index: 365. (index mismatch)
I20260808 01:51:31.643878 5210 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 3.360, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.645174 7368 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20260808 01:51:31.646420 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.648131 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.648499 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.647357 5207 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 2.366, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.650445 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 1.408, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.653499 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 402. Preceding OpId from leader: term: 2 index: 403. (index mismatch)
I20260808 01:51:31.655205 7323 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I20260808 01:51:31.655548 5210 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 408. Preceding OpId from leader: term: 1 index: 409. (index mismatch)
I20260808 01:51:31.657521 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I20260808 01:51:31.659276 5069 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 360. Preceding OpId from leader: term: 3 index: 361. (index mismatch)
I20260808 01:51:31.661082 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 366. Preceding OpId from leader: term: 2 index: 367. (index mismatch)
I20260808 01:51:31.660733 5073 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.664388 7339 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I20260808 01:51:31.665594 7338 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:31.665779 7404 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.667531 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 368: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.672101 7406 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.674546 4786 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.675715 4786 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.676813 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.678622 7326 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.409: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.683655 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.365: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.687009 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.409: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.689424 7282 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.692395 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.361: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.692713 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.699213 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.701901 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.702992 5207 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.403: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.705847 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 408: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.706574 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 402: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.715837 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.716487 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 364: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.717083 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 360: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.727240 5207 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 2.369, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.731164 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 1.405, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.734867 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.735240 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.735323 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
I20260808 01:51:31.736603 7406 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20260808 01:51:31.738042 4779 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 402 to 403, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 403 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.740550 5210 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 1.371, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.742044 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.741266 5208 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 2.373, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.743966 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.745441 5211 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.749075 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 373. Preceding OpId from leader: term: 2 index: 374. (index mismatch)
I20260808 01:51:31.750383 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.750735 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.750833 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20260808 01:51:31.751519 7282 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20260808 01:51:31.752115 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20260808 01:51:31.752877 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 405: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.755039 4779 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.756810 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 364 to 365, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.759577 4779 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 360 to 361, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 361 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.761382 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.761756 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.762064 7368 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.767714 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 371. Preceding OpId from leader: term: 1 index: 372. (index mismatch)
I20260808 01:51:31.769795 7282 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.001s
I20260808 01:51:31.771860 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.370: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.780781 7404 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.786255 5069 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.790725 7367 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.793599 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 369: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.795269 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.809530 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 371: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.817698 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 373: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.826565 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 2.403, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.828217 5069 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 409, Committed index: 409, Last appended: 1.409, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.830950 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 2.365, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.831101 4779 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 369 to 370, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.832262 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.835256 5211 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 409. Preceding OpId from leader: term: 1 index: 410. (index mismatch)
I20260808 01:51:31.837162 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:31.837786 5075 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 365. Preceding OpId from leader: term: 2 index: 366. (index mismatch)
I20260808 01:51:31.838636 7323 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 410, Last known committed idx: 409, Time since last communication: 0.000s
W20260808 01:51:31.837538 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.841539 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.841918 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.843374 7404 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20260808 01:51:31.842862 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:31.846454 7326 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.410: config changed from index 409 to 410, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.845515 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.851490 7282 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.855497 5211 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 2.367, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.861769 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.366: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.856817 5210 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.410: config changed from index 409 to 410, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.864625 5073 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 1.406, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.870610 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:31.870674 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 365: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260808 01:51:31.870957 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.873307 5208 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20260808 01:51:31.875856 7323 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20260808 01:51:31.876036 5207 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 361, Committed index: 361, Last appended: 3.361, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:31.880573 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 403. Preceding OpId from leader: term: 2 index: 404. (index mismatch)
I20260808 01:51:31.882124 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:31.882043 7366 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
W20260808 01:51:31.882614 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.882665 5075 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 361. Preceding OpId from leader: term: 3 index: 362. (index mismatch)
I20260808 01:51:31.883298 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:31.883703 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:31.884110 7339 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 362, Last known committed idx: 361, Time since last communication: 0.000s
I20260808 01:51:31.884415 5074 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 367. Preceding OpId from leader: term: 2 index: 368. (index mismatch)
I20260808 01:51:31.887019 7367 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:31.894961 7366 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.897066 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 409: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.897831 7339 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.898662 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 365 to 366, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.902478 4779 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 409 to 410, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.905007 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.404: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.909272 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 403: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:31.909672 5075 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.362: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.913938 7406 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.407: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.918291 5208 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.923934 7367 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.927371 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 361: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.928154 5075 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 410, Committed index: 410, Last appended: 1.410, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.929663 5074 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.937933 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 406: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.940078 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:31.943310 5208 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 2.370, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.943590 5211 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.946971 5207 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 2.374, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.951220 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 2.366, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:31.954133 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 374. Preceding OpId from leader: term: 2 index: 375. (index mismatch)
I20260808 01:51:31.955120 5069 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 370. Preceding OpId from leader: term: 2 index: 371. (index mismatch)
I20260808 01:51:31.955701 7368 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260808 01:51:31.958415 7368 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20260808 01:51:31.961062 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 366. Preceding OpId from leader: term: 2 index: 367. (index mismatch)
I20260808 01:51:31.962450 5207 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 410. Preceding OpId from leader: term: 1 index: 411. (index mismatch)
I20260808 01:51:31.963771 7326 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 411, Last known committed idx: 410, Time since last communication: 0.000s
I20260808 01:51:31.966816 7282 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20260808 01:51:31.968786 4786 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 406 to 407, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:31.969818 7368 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.963768 5074 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20260808 01:51:31.973274 5073 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.375: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.974206 7406 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.411: config changed from index 410 to 411, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.975572 4786 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.977458 7404 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.979375 4786 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 361 to 362, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 362 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:31.981732 7378 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20260808 01:51:31.982687 5075 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.371: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.984046 4786 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 403 to 404, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 404 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.983958 7378 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:31.992192 5207 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.411: config changed from index 410 to 411, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.002853 5073 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.367: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.007086 5074 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 407, Committed index: 407, Last appended: 1.407, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:32.008298 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 410: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.010020 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.014863 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 370: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.017174 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20260808 01:51:32.017971 5074 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 2.404, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.018711 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.001s
I20260808 01:51:32.024595 7326 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.408: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.025260 7404 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.030723 5010 consensus_queue.cc:579] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.031108 5010 consensus_peers.cc:597] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/86d58fdfa95b45f68c0648a585924d05/.kudutmp.newsegmentTmC4ks (347480064 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.035955 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.036453 5208 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 2.368, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:32.037254 5207 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 404. Preceding OpId from leader: term: 2 index: 405. (index mismatch)
I20260808 01:51:32.038197 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 407: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.044569 5208 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 3.362, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:32.047629 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5843450fd1f54792899cc7190c94821f with cas_config_opid_index 372: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.047343 5074 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.049484 7406 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20260808 01:51:32.041200 4782 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 410 to 411, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 411 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:32.052973 5073 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 362. Preceding OpId from leader: term: 3 index: 363. (index mismatch)
I20260808 01:51:32.054229 5075 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 368. Preceding OpId from leader: term: 2 index: 369. (index mismatch)
I20260808 01:51:32.058540 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20260808 01:51:32.058677 7339 consensus_queue.cc:1048] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20260808 01:51:32.061221 7326 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.052539 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 374: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.070331 7339 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.064523 4782 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 374 to 375, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.070703 5208 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.405: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.072192 4782 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 370 to 371, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.075793 5073 raft_consensus.cc:2955] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.369: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.073696 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 404: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.087309 7404 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.090160 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8d367581a54c433ea4f227bd4f479ed2 with cas_config_opid_index 368: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.091574 5073 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.363: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.096483 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1fff9439a0bd4286827363f69521c44f with cas_config_opid_index 362: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.097301 4779 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 407 to 408, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.099013 4779 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 366 to 367, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:32.101181 4779 catalog_manager.cc:5697] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 372 to 373, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 1 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:32.107414 5073 consensus_queue.cc:237] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 411, Committed index: 411, Last appended: 1.411, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.112885 5010 consensus_queue.cc:579] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
I20260808 01:51:32.116268 5208 raft_consensus.cc:1275] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 411. Preceding OpId from leader: term: 1 index: 412. (index mismatch)
W20260808 01:51:32.113334 5010 consensus_peers.cc:597] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/9ee6f0c29217471e93394d4b9814144e/.kudutmp.newsegmentck793C (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.117806 7406 consensus_queue.cc:1048] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 412, Last known committed idx: 411, Time since last communication: 0.000s
I20260808 01:51:32.136765 4784 catalog_manager.cc:5697] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 368 to 369, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.139446 4784 catalog_manager.cc:5697] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 404 to 405, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 405 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:32.144356 7406 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.412: config changed from index 411 to 412, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.147374 5208 raft_consensus.cc:2955] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.412: config changed from index 411 to 412, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.141243 4784 catalog_manager.cc:5697] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 362 to 363, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New cstate: current_term: 3 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.163874 5210 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 2.375, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.168625 5210 consensus_queue.cc:237] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 2.367, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.170256 5073 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 375. Preceding OpId from leader: term: 2 index: 376. (index mismatch)
I20260808 01:51:32.171559 7378 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s
I20260808 01:51:32.177776 5073 raft_consensus.cc:1275] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 367. Preceding OpId from leader: term: 2 index: 368. (index mismatch)
I20260808 01:51:32.177946 7404 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.179165 7339 consensus_queue.cc:1048] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20260808 01:51:32.183658 5074 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.376: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.188298 7339 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.191903 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7aa70246b445149411741e595f27c2 with cas_config_opid_index 375: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.194212 5210 consensus_queue.cc:237] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 2.371, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.195094 5075 raft_consensus.cc:2955] T 0c44eefb43024e9290fdba8647e745fe P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.368: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.206126 5149 consensus_queue.cc:579] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.231437 5149 consensus_peers.cc:597] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2d7aa70246b445149411741e595f27c2/.kudutmp.newsegmentvERCpp (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.232352 5149 consensus_queue.cc:579] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.232740 5149 consensus_peers.cc:597] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/0c44eefb43024e9290fdba8647e745fe/.kudutmp.newsegmentfdcov2 (347557888 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.235172 5207 consensus_queue.cc:237] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 1.373, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.239269 5075 raft_consensus.cc:1275] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 371. Preceding OpId from leader: term: 2 index: 372. (index mismatch)
I20260808 01:51:32.241106 7378 consensus_queue.cc:1048] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.001s
I20260808 01:51:32.241585 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c44eefb43024e9290fdba8647e745fe with cas_config_opid_index 367: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.243516 5149 consensus_queue.cc:579] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.243888 5149 consensus_peers.cc:597] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/24ad2932d9a1489b96a63bc762f6a23f/.kudutmp.newsegmentdKFiv0 (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.246953 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ee6f0c29217471e93394d4b9814144e with cas_config_opid_index 411: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.250240 6765 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:32.249838 7378 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.250327 5069 consensus_queue.cc:237] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 1.408, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.252409 5074 raft_consensus.cc:2955] T 24ad2932d9a1489b96a63bc762f6a23f P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.372: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.256278 5073 raft_consensus.cc:1275] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20260808 01:51:32.257951 7368 consensus_queue.cc:1048] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20260808 01:51:32.258888 5207 raft_consensus.cc:1275] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 1 index: 408. Preceding OpId from leader: term: 1 index: 409. (index mismatch)
I20260808 01:51:32.258903 5075 consensus_queue.cc:237] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 2.405, Last appended by leader: 204, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:32.261026 6530 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260808 01:51:32.264053 5010 consensus_queue.cc:579] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.267478 5010 consensus_peers.cc:597] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/2a97be89f618431b90b49c97b2c9f001/.kudutmp.newsegmentUaJIzO (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.269411 5210 raft_consensus.cc:1275] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Refusing update from remote peer ec65d57b7135430eb6695032355bd356: Log matching property violated. Preceding OpId in replica: term: 2 index: 405. Preceding OpId from leader: term: 2 index: 406. (index mismatch)
I20260808 01:51:32.271325 7326 consensus_queue.cc:1048] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20260808 01:51:32.271641 4779 catalog_manager.cc:5697] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 375 to 376, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 376 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:32.277730 7326 consensus_queue.cc:1048] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I20260808 01:51:32.286319 7406 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P ec65d57b7135430eb6695032355bd356 [term 2 LEADER]: Committing config change with OpId 2.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.292851 5211 raft_consensus.cc:2955] T 86d58fdfa95b45f68c0648a585924d05 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 FOLLOWER]: Committing config change with OpId 2.406: config changed from index 405 to 406, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.306253 7323 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 [term 1 LEADER]: Committing config change with OpId 1.409: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.311298 5210 raft_consensus.cc:2955] T 2a97be89f618431b90b49c97b2c9f001 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 FOLLOWER]: Committing config change with OpId 1.409: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.321920 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24ad2932d9a1489b96a63bc762f6a23f with cas_config_opid_index 371: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.324545 5149 consensus_queue.cc:579] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.325174 5149 consensus_peers.cc:597] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/5843450fd1f54792899cc7190c94821f/.kudutmp.newsegment3w4r2B (347512832 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.331203 5212 consensus_queue.cc:237] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 2.376, Last appended by leader: 203, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } }
I20260808 01:51:32.335883 4770 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 86d58fdfa95b45f68c0648a585924d05 with cas_config_opid_index 405: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:51:32.337926 4770 catalog_manager.cc:5210] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a97be89f618431b90b49c97b2c9f001 with cas_config_opid_index 408: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260808 01:51:32.324846 4784 catalog_manager.cc:5697] T 9ee6f0c29217471e93394d4b9814144e P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 411 to 412, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.339841 5206 consensus_queue.cc:237] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 2.369, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.344633 7368 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 1 LEADER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.346092 5205 consensus_queue.cc:237] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 3.363, Last appended by leader: 205, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } }
I20260808 01:51:32.355149 5149 consensus_queue.cc:579] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.355530 5149 consensus_peers.cc:597] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/1fff9439a0bd4286827363f69521c44f/.kudutmp.newsegmentXMxzRd (347496448 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.356781 5074 raft_consensus.cc:1275] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 363. Preceding OpId from leader: term: 3 index: 364. (index mismatch)
I20260808 01:51:32.358044 5054 tablet_service.cc:1467] Tablet server has 4 leaders and 0 scanners
I20260808 01:51:32.358807 5074 raft_consensus.cc:1275] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 376. Preceding OpId from leader: term: 2 index: 377. (index mismatch)
I20260808 01:51:32.359326 7282 consensus_queue.cc:1048] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s
I20260808 01:51:32.360251 7339 consensus_queue.cc:1048] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I20260808 01:51:32.354486 5073 raft_consensus.cc:2955] T 5843450fd1f54792899cc7190c94821f P ec65d57b7135430eb6695032355bd356 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.366822 5149 consensus_queue.cc:579] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [LEADER]: The tablet replica hosted on peer c37035739bf347d49e5ed4b2f884adac has failed
W20260808 01:51:32.367184 5149 consensus_peers.cc:597] T 8d367581a54c433ea4f227bd4f479ed2 P 99d0ee4efbd948a3b143ff2297ad9fc9 -> Peer c37035739bf347d49e5ed4b2f884adac (127.29.180.68:35651): Couldn't send request to peer c37035739bf347d49e5ed4b2f884adac. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/dist-test-taskz8giuv/test-tmp/disk_failure-itest.0.TabletServerDiskError_TabletServerDiskErrorITest.TestFailOnBootstrap_0.1786153668592116-30417-0/minicluster-data/ts-3/wal/wals/8d367581a54c433ea4f227bd4f479ed2/.kudutmp.newsegment0YFxLd (347529216 bytes available vs 518351011 bytes reserved) (error 28). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:32.370046 4784 catalog_manager.cc:5697] T 0c44eefb43024e9290fdba8647e745fe P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 367 to 368, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:32.373273 5069 raft_consensus.cc:1275] T 8d367581a54c433ea4f227bd4f479ed2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Refusing update from remote peer 99d0ee4efbd948a3b143ff2297ad9fc9: Log matching property violated. Preceding OpId in replica: term: 2 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20260808 01:51:32.374114 4784 catalog_manager.cc:5697] T 24ad2932d9a1489b96a63bc762f6a23f P 99d0ee4efbd948a3b143ff2297ad9fc9 reported cstate change: config changed from index 371 to 372, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 2 leader_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
I20260808 01:51:32.374394 7404 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 3 LEADER]: Committing config change with OpId 3.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.374894 7282 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P 99d0ee4efbd948a3b143ff2297ad9fc9 [term 2 LEADER]: Committing config change with OpId 2.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.387444 5069 raft_consensus.cc:2955] T 1fff9439a0bd4286827363f69521c44f P ec65d57b7135430eb6695032355bd356 [term 3 FOLLOWER]: Committing config change with OpId 3.364: config changed from index 363 to 364, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } peers { permanent_uuid: "c37035739bf347d49e5ed4b2f884adac" member_type: NON_VOTER last_known_addr { host: "127.29.180.68" port: 35651 } attrs { promote: true } } }
I20260808 01:51:32.387907 5075 raft_consensus.cc:2955] T 2d7aa70246b445149411741e595f27c2 P ec65d57b7135430eb6695032355bd356 [term 2 FOLLOWER]: Committing config change with OpId 2.377: config changed from index 376 to 377, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67" port: 46093 } } }
I20260808 01:51:32.393371 4782 catalog_manager.cc:5697] T 2a97be89f618431b90b49c97b2c9f001 P ec65d57b7135430eb6695032355bd356 reported cstate change: config changed from index 408 to 409, NON_VOTER c37035739bf347d49e5ed4b2f884adac (127.29.180.68) added. New cstate: current_term: 1 leader_uuid: "ec65d57b7135430eb6695032355bd356" committed_config { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "ec65d57b7135430eb6695032355bd356" member_type: VOTER last_known_addr { host: "127.29.180.66" port: 43479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99d0ee4efbd948a3b143ff2297ad9fc9" member_type: VOTER last_known_addr { host: "127.29.180.67"